Elamur Research⌕
☰
home / papers / 2026 / CityVG: Contrastive Fine-Tuning and Reward-Based Chain-of-Thought Reasoning for Zero-Shot City-Scale 3D Visual Grounding

CityVG: Contrastive Fine-Tuning and Reward-Based Chain-of-Thought Reasoning for Zero-Shot City-Scale 3D Visual Grounding

Paper · 2026 · Jianjun Zhang and Hanli Wang

Summary

The paper addresses city-scale 3D visual grounding: finding a specific object in a huge urban point cloud from a natural-language description. Earlier city-level systems such as CityRefer and CityAnchor depend on costly manual annotations. CityVG is presented as the first zero-shot framework for this task, and it works in two phases.

  1. Instance extraction. Mask3D, combined with a VLM for open-vocabulary labelling, splits the scene into object instances. Each instance is rendered from overhead and oblique viewpoints.
  2. Retrieval. A Visual Pairwise Scene Graph links instances to nearby landmarks, and a VLM turns these links into relational captions. LLM-driven clustering then mines positive and negative pairs automatically, which are used to contrastively fine-tune a BGE text encoder. The tuned encoder narrows thousands of objects down to a short candidate list.
  3. Reasoning. A VLM runs several Chain-of-Thought trajectories in parallel. It sums rewards for class, appearance, and spatial consistency to pick the target.

Without any grounding labels, CityVG approaches supervised accuracy and runs faster than CityAnchor. This makes annotation-free urban geo-spatial analysis more practical.

Key points

  • The system is evaluated on the CityRefer and CityAnchor benchmarks. The full zero-shot model reaches 44.13% overall Acc@0.50 in the ablation study.

  • A supervised variant (CityVG†) exceeds the fully supervised CityAnchor, scoring 55.38 versus 50.69 in the first reported metric column.

  • The contrastive fine-tuning stage improves candidate recall:

    • CR@1 rises from 31.58 to 38.95.
    • CR@5 rises from 55.79 to 66.05.
    • Swapping LLM-driven clustering for K-Means, or removing contrastive learning, lowers recall.

  • The default settings are 8 retrieved candidates and 3 reasoning trajectories. Adding more trajectories brings only small gains.

  • Inference time per query:

    MethodCityReferCityAnchor
    CityVG29.47 s49.65 s
    CityAnchor32.45 s51.72 s
    CityRefer42.27 s98.91 s

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksVision & language · 811 works

Cite

Cite as Jianjun Zhang and Hanli Wang. 2026. CityVG: Contrastive Fine-Tuning and Reward-Based Chain-of-Thought Reasoning for Zero-Shot City-Scale 3D Visual Grounding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1617

@inproceedings{zhang2026cityvg,
  title = {{CityVG: Contrastive Fine-Tuning and Reward-Based Chain-of-Thought Reasoning for Zero-Shot City-Scale 3D Visual Grounding}},
  author = {Jianjun Zhang and Hanli Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1617},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1617}
}

References

49 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 2019. 3D Scene Graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5664–5673. find
  2. Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 2016. 3D semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1534–1543. find
  3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.
  4. Ricardo Garcia, Shizhe Chen, and Cordelia Schmid. 2025. Towards Generalizable Vision-Language Robotic Manipulation: A benchmark and llm-guided 3d policy. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 8996– 9002. find
  5. Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, and Rama Chellappa. 2024. ConceptGraphs: Open-vocabulary 3d scene graphs for perception and planning. In Proceedings of the IEEE International Conference on R… find
  6. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 14852–14882.
  7. Juan Pablo Hourcade, Tristan Braud, Peng Yuan Zhou, Lin Wang, Dianlei Xu, Zijun Lin, Abhishek Kumar, Carlos Bermejo, and Pan Hui. 2024. All One Needs to Know about Metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda. Foundations and Trends in Human– Comp… find
  8. Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham. 2022. SensatUrban: Learning semantics from urban-scale photogrammet-ric point clouds. International Journal of Computer Vision, 130(2):316–343. find
  9. Tianxiang Hu, Pei Zhang, Baosong Yang, Jun Xie, Derek F. Wong, and Rui Wang. 2024. Large language model for multi-domain translation: Benchmarking and domain CoT fine-tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5726–5746. find
  10. Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. 2022. Bottom up top down detection transformers for language grounding in images and point clouds. In European Conference on Computer Vision, pages 417–433. find
  11. Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. 2020. ScanRefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, pages 202–221. find
  12. Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. 2023. LERF: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729–19739. find
  13. Meida Chen, Qingyong Hu, Zifan Yu, Hugues THOMAS, Andrew Feng, Yu Hou, Kyle McCullough, Fengbo Ren, and Lucio Soibelman. 2022. STPLS3D: A large-scale synthetic and real aerial photogrammetry 3d point cloud dataset. In The British Machine Vision Conference. find
  14. Ue-Hwan Kim, Jin-Man Park, Taek-Jin Song, and Jong-Hwan Kim. 2019. 3-D Scene Graph: A sparse and semantic representation of physical environments for intelligent agents. IEEE Transactions on Cybernetics, 50(12):4921–4933. find
  15. Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5828–5839. find
  16. Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. 2024. Open3DSG: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 141… find
  17. Ping Kong, Ruonan Liu, Zongxia Xie, and Zhibo Pang. 2025. VLN-KHVR: Knowledge-and-history aware visual representation for continuous vision-and-language navigation. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 5236–5243. find
  18. Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, and Iryna Gurevych. 2025. Fine-tuning on diverse reasoning chains drives within-inference CoT refinement in LLMs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 3789–3808. find
  19. Vignesh Kothapalli, Hamed Firooz, and Maziar Sanjabi. 2025. CoT-ICL lab: A synthetic framework for studying chain-of-thought learning from in-context demonstrations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 14620–14642. find
  20. Zhipeng Qian, Yiwei Ma, Zhekai Lin, Jiayi Ji, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. 2024. Multi-branch collaborative learning network for 3d visual grounding. In European Conference on Computer Vision, pages 381–398. find
  21. Jinpeng Li, Haiping Wang, Yuan Liu, Zhiyang Dou, Yuexin Ma, Sibei Yang, Yuan Li, Wenping Wang, Zhen Dong, and Bisheng Yang. 2025a. CityAnchor: City-scale 3d visual grounding with multi-modality llms. In International Conference on Learning Representations. find
  22. Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, and Junwei Liang. 2025b. SeeGround: See and ground for zero-shot open-vocabulary 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3707– 3717. find
  23. Liqiang Lin, Yilin Liu, Yue Hu, Xingguang Yan, Ke Xie, and Hui Huang. 2022. Capturing, reconstructing, and simulating: The urbanscene3d dataset. In European Conference on Computer Vision, pages 93–109. find
  24. Sergey Linok, Tatiana Zemskova, Svetlana Ladanova, Roman Titkov, Dmitry Yudin, Maxim Monastyrny, and Aleksei Valenkov. 2025. Beyond Bare Queries: Open-vocabulary object grounding with 3d scene graph. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 13582– 13589. find
  25. Zheng Liu, Jianjun Zhang, Ming Zhang, Runze Ke, Chengcheng Yu, and Ligang Liu. 2026. Unsupervised point cloud reconstruction via recurrent multi-step moving strategy. IEEE Transactions on Multimedia, 28:972–984. find
  26. Taiki Miyanishi, Fumiya Kitamori, Shuhei Kurita, Jung-dae Lee, Motoaki Kawanabe, and Nakamasa Inoue. 2023. CityRefer: Geography-aware 3d visual grounding dataset on city-scale point cloud data. In Advances in Neural Information Processing Systems, volume 36, pages 77758–77770. find
  27. Hoang Nguyen, Ye Liu, Chenwei Zhang, Tao Zhang, and Philip Yu. 2023. CoF-CoT: Enhancing large language models with coarse-to-fine chain-of-thought prompting for multi-domain NLU tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 12109–12119. find
  28. Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. 2023. OpenScene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 815–824. find
  29. Michael J. Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, and Diyi Yang. 2025. Synthe-sizeMe! inducing persona-guided prompts for personalized reward models in LLMs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 8045–8078. find
  30. Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. 2023. Mask3D: Mask transformer for 3d semantic instance segmentation. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 8216–8223. find
  31. Hai-Long Sun, Zhun Sun, Houwen Peng, and Han-Jia Ye. 2025. Mitigating visual forgetting via take-along visual conditioning for multi-modal long CoT reasoning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 5158–5171.
  32. Ozan Unal, Christos Sakaridis, Suman Saha, and Luc Van Gool. 2024. Four ways to improve verbo-visual fusion for dense 3d visual grounding. In European Conference on Computer Vision, pages 196–213. find
  33. Yuan Wang, Yali Li, and Shengjin Wang. 2024. G3-LQ: Marrying hyperbolic alignment with explicit semantic-geometric modeling for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13917–13926. find
  34. Zehan Wang, Haifeng Huang, Yang Zhao, Linjun Li, Xize Cheng, Yichen Zhu, Aoxiong Yin, and Zhou Zhao. 2023a. Distilling coarse-to-fine semantic matching knowledge for weakly supervised 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2662–2671. find
  35. Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang, and Lu Sheng. 2023b. VL-SAT: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21560–21569. find
  36. Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, and Ge Yu. 2025. RankCoT: Refining knowledge for retrieval-augmented generation through ranking chain-of-thoughts. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 12857– 1… find
  37. Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. 2023. EDA: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19231–19242. find
  38. Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. 2026. Toward Visual Grounding: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3):2749– 2771. find
  39. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-Pack: Packed resources for general chinese embeddings. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 641–649.
  40. Zhongyuan Wang, Shanghang Zhang, and Renjing Xu. 2025. MapNav: A novel memory representation via annotated semantic maps for vision-and-language navigation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 13032–13056. find
  41. Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 2021. 3DVG-Transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928–2937. find
  42. Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 2023. 3D-VisTA: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921. find
  43. Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2025a. VLM-Grounder: A vlm agent for zero-shot 3d visual grounding. In Proceedings of the Conference on Robot Learning, volume 270, pages 3961–3985. find
  44. Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. 2025b. SoftCoT: Soft chain-of-thought for efficient reasoning with LLMs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 23336–23351.
  45. Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. 2024. LLM-Grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 7694– 7701. find
  46. Zhihao Yuan, Jinke Ren, Chun-Mei Feng, Hengshuang Zhao, Shuguang Cui, and Zhen Li. 2024. Visual programming for zero-shot open-vocabulary 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20623–20633. find
  47. Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. 2021. InstanceRefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference on Computer… find
  48. Jianjun Zhang, Zhipeng Jiang, Qinjun Qiu, and Zheng Liu. 2024. TCFAP-Net: Transformer-based cross-feature fusion and adaptive perception network for large-scale point cloud semantic segmentation. Pattern Recognition, 154:110630. find
  49. Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-cityvg-contrastive-fine-tuning-reward/