Elamur Research⌕
☰
home / papers / 2026 / VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval

VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval

Paper · 2026 · Di Wu, Yixin Wan and Kai-Wei Chang

Summary

Text-to-image retrieval often fails when a query demands specific visual configurations, such as an animal's posture or the camera angle. Cross-modal embeddings tend to register which concepts appear but not how they are arranged. The authors propose Visualize-then-Retrieve (VisRet), a training-free pipeline with two stages:

  1. An LLM turns the textual query into an image-generation instruction, and a T2I generation model renders one or more synthetic images from it.
  2. These images serve as queries for image-to-image search, and the ranked lists are merged.

Because matching happens inside the visual modality, the method avoids weak text-image alignment and works with off-the-shelf retrievers and precomputed indices. The authors evaluate it with CLIP and E5-V on Visual-RAG, INQUIRE-Rerank, Microsoft COCO and a newly built multi-entity benchmark, Visual-RAG-ME. VisRet beats both direct cross-modal matching and text-rewriting baselines on retrieval quality. It also raises downstream visual question answering accuracy in a RAG setup. This makes it a simple, modular way to strengthen knowledge-intensive multimodal systems.

Key points

  • Average nDCG@30 gains across the four benchmarks are 0.125 with CLIP and 0.121 with E5-V. Harder subsets INQUIRE-Rerank-Hard and COCO-Hard were used.
  • Downstream VQA accuracy rises on Visual-RAG / Visual-RAG-ME:
    • +3.8% / +15.7% with top-1 retrieval;
    • +3.9% / +11.1% with top-10 retrieval.
  • Visual-RAG-ME was built as follows:
    • BM25 over scientific names finds related second entities for Visual-RAG organisms.
    • The authors manually reviewed all 391 original questions, producing 82 comparison questions.
    • Questions answerable without images by GPT-4o and GPT-4o-mini, and topical duplicates, were filtered out, leaving 50 questions.
    • Annotations are released under CC BY-NC 4.0.
  • Results from several generated images are combined with Reciprocal Rank Fusion. Using a single visualization costs only a little performance.
  • The method is robust to the choice of instruction LLM, retriever and reader LVLM, but depends strongly on the T2I generator; stronger generators bring larger gains. Remaining errors come from flawed instructions, poor visualizations, and retrievers being distracted by background or appearance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksInformation retrieval & search · 473 works

Cite

Cite as Di Wu, Yixin Wan and Kai-Wei Chang. 2026. VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1192

@inproceedings{wu2026visret,
  title = {{VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval}},
  author = {Di Wu, Yixin Wan and Kai-Wei Chang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1192},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1192}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Black Forest Labs. 2024. Announcing black forest labs. https://bfl.ai/blog/24-08-01-bfl. News, published August 1, 2024. ↗
  2. Tobias Braun, Mark Rothermel, Marcus Rohrbach, and Anna Rohrbach. 2024. DEFAME: dynamic evidence-based fact-checking with multimodal experts. CoRR, abs/2412.10510. — [arXiv:2412.10510](https://arxiv.org/abs/2412.10510) find
  3. Yingshan Chang, Guihong Cao, Mridu Narang, Jianfeng Gao, Hisami Suzuki, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal QA. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16474–16483. IEEE. find
  4. Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W. Cohen. 2022. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emir…
  5. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325.
  6. Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, US…
  7. Guofeng Ding, Yiding Lu, Peng Hu, Mouxing Yang, Yijie Lin, and Xi Peng. 2025. Visual abstraction: A plug-and-play approach for text-visual retrieval. In Forty-second International Conference on Machine Learning. find
  8. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-firs… find
  9. Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. VSE++: improving visual-semantic embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 12. BMVA Press. find
  10. Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman H. Khan, Wangmeng Zuo, Rick Siow Mong Goh, and Yong Liu. 2025. VQA4CIR: boosting composed image retrieval with visual question answering. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 -March 4, 2… find
  11. Alisa Fortin, Guillaume Vernade, Kat Kampf, and Ammaar Reshi. 2025. Introducing gemini 2.5 flash image, our state-of-the-art image model. https://developers.googleblog.com/en/ introducing-gemini-2-5-flash-image/. Google Developers Blog; posted Aug. 26, 2025. ↗
  12. Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomás Mikolov. 2013. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013.… find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. 2017. Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, pages 1463–1471. find
  15. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Intell. Res., 47:853–899. find
  16. Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge J. Belongie, and Oisin Mac Aodha. 2021. Benchmarking representation learning for natural world image collections. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 12884–12893. Co… find
  17. Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. 2024. Mrag-bench: Vision-centric evaluation for retrieval-augmented multimodal models. CoRR, abs/2410.08182. — [arXiv:2410.08182](https://arxiv.org/abs/2410.08182) find
  18. Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-V: universal embeddings with multimodal large language models. CoRR, abs/2407.12580. — [arXiv:2407.12580](https://arxiv.org/abs/2407.12580) find
  19. Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. Text encoders bottleneck compositionality in contrastive vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4933–4944. Association for… find
  20. Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539.
  21. Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. 2025. CLIP behaves like a bag-of-words model cross-modally but not uni-modally. CoRR, abs/2502.03566. — [arXiv:2502.03566](https://arxiv.org/abs/2502.03566) find
  22. Saehyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, and Sungroh Yoon. 2024. Interactive text-to-image retrieval with large language models: A plug-and-play approach. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, T… find
  23. Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. 2023. Chatting makes perfect: Chat-based image retrieval. Advances in Neural Information Processing Systems, 36:61437–61449. find
  24. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings…
  25. Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua. 2024. Generative cross-modal retrieval: Memorizing images in multimodal language models for retrieval and beyond. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  26. Zheyuan Liu, Weixuan Sun, Damien Teney, and Stephen Gould. 2024. Candidate set re-ranking for composed image retrieval with dual multi-modal encoder. find
  27. Trans. Mach. Learn. Res., 2024. find
  28. OpenAI. 2023a. Dall·e 3. https://openai.com/ index/dall-e-3/. Accessed: 2025-05-18. ↗
  29. OpenAI. 2023b. Gpt-4 technical report. abs/2303.08774.
  30. ArXiv, OpenAI. 2025. Gpt image 1. https://platform. Ac-openai.com/docs/models/gpt-image-1. cessed: 2025-05-18. find
  31. Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, and Tat-Seng Chua. 2025. Tiger: Unifying text-to-image generation and retrieval with large multimodal models. find
  32. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  33. S. E. Robertson and S. Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR ’94, pages 232–241, London. Springer London. find
  34. Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. 2023. knn-diffusion: Image generation via large-scale retrieval. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  35. Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel J. Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn. 2024. INQUIRE: A natural world text-to-image retrieval benchmark. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information P… find
  36. Xiaodan Wang, Chengyu Wang, Lei Li, Zhixu Li, Ben Chen, Linbo Jin, Jun Huang, Yanghua Xiao, and Ming Gao. 2023. Fashionklip: Enhancing ecommerce image-text retrieval with fashion multimodal conceptual knowledge graph. In Proceedings of the The 61st Annual Meeting of the Association for Computational… find
  37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  38. Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang… — Qwen-Image Technical Report
  39. Yin Wu, Quanyu Long, Jing Li, Jianfei Yu, and Wenya Wang. 2025b. Visual-rag: Benchmarking text-to-image retrieval augmented generation for visual knowledge intensive queries. CoRR, abs/2502.16636. — [arXiv:2502.16636](https://arxiv.org/abs/2502.16636) find
  40. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67–78. find
  41. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models.
  42. Trans. Mach. Learn. Res., 2022. find
  43. Mert Yüksekgönül, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-visret-visualization-improves-knowledge-intensive/