Elamur Research⌕
☰
home / papers / 2026 / GeoRC: A Benchmark for Geolocation Reasoning Chains

GeoRC: A Benchmark for Geolocation Reasoning Chains

Paper · 2026 · Mohit Talreja, Joshua Diao, Jim James et al.

Summary

GeoRC addresses a gap between how well vision-language models (VLMs) guess where a photo was taken and how well they justify those guesses. Frontier VLMs now locate images about as accurately as top humans, but their explanations often cite evidence that is absent or irrelevant. The authors built a benchmark of 800 expert-written reasoning chains covering 500 GeoGuessr scenes. Three Champion-tier players wrote the chains under a shared protocol, and one of them is the 2025 World Cup winner. Candidate chains are scored against the expert references by precision, recall and F1. The authors first had humans grade candidates for calibration, then compared several LLM-as-a-judge and VLM-as-a-judge schemes. A Qwen 3 LLM judge agreed best with the human experts. Large closed models still trail the experts by roughly 12 points. Small open-weight models do only marginally better than a text-only baseline that invents reasoning from the known location without seeing the image. The results suggest current vision encoders struggle to pick up fine, low-pixel details. The benchmark offers a way to measure auditable, trustworthy geolocation reasoning.

Key points

  • Dataset: 100 GeoGuessr challenges with 5 locations each (500 scenes), drawn from maps such as An Arbitrary World. All three experts annotated the same 150 locations, and they split the remaining 350 among themselves.
  • Reasoning content: Chains cite discriminative attributes such as infrastructure (the most frequent category), vegetation, architecture, language, Street View "meta" cues, geology and culture (the least frequent).
  • Evaluation method: Experts graded 150 candidate chains using one-to-all bipartite matching, never grading their own chains. Among the automated judges tested, Qwen 3 as LLM-as-a-judge tracked these human scores most closely.
  • Error types: The dominant VLM failures are misattribution, hallucination, false tool use, axiomatic irrelevance and missed fine details.
  • Release and limitations: Chains are released under CC BY and code under Apache, along with scripts to re-download the images. The chains are English-only, all models received a single fixed prompt, and the largest open-weight models were not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Mohit Talreja, Joshua Diao, Jim James, Radu Casapu, Tejas Santanam, Ethan Mendes, Alan Ritter, Wei Xu and James Hays. 2026. GeoRC: A Benchmark for Geolocation Reasoning Chains. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1883

@inproceedings{talreja2026georc,
  title = {{GeoRC: A Benchmark for Geolocation Reasoning Chains}},
  author = {Mohit Talreja, Joshua Diao, Jim James, Radu Casapu, Tejas Santanam, Ethan Mendes, Alan Ritter, Wei Xu and James Hays},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1883},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1883}
}

References

42 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gemini Team, Google DeepMind. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261.
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. Preprint, a…
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. Preprint, arXiv:2502.1…
  4. Fenghua Cheng, Jinxiang Wang, Sen Wang, Zi Huang, and Xue Li. 2025. Geoguess: Multimodal reasoning based on hierarchy of visual information in street view. Preprint, arXiv:2506.16633. — [arXiv:2506.16633](https://arxiv.org/abs/2506.16633) find
  5. Brandon Clark, Anthony Kerrigan, Parth Parag Kulkarni, Vicente Vivanco Cepeda, and Mubarak Shah. 2023. Where we are and what we’re looking at: Query based worldwide image geo-localization using hierarchies and scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti… find
  6. Condé Nast Traveler. 2011. Condé Nast Traveler: Where Are You? Assouline Publishing, New York. Introduction by Klara Glowczewska. find
  7. Google DeepMind. 2025. Gemini 3: State-of-the-art multimodal reasoning and agentic intelligence. Google Blog. Released November 18, 2025. Accessed: April 20, 2026. find
  8. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, and 31 others. 2025. M… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  10. Yoshinari Fujinuma. 2025. Contrastive decoding mitigates score range bias in llm-as-a-judge. Preprint, arXiv:2510.18196. find
  11. GeoGuessr AB. 2013. Geoguessr: A geography guessing game. https://www.geoguessr.com/. Accessed: April 20, 2026. ↗
  12. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. 2026. A survey on llm-as-a-judge. The Innovation, page 101253. find
  13. Lukas Haas, Silas Alberti, and Michal Skreta. 2023a. Learning generalized zero-shot learners for open-domain image geolocalization. Preprint, arXiv:2302.00275. — [arXiv:2302.00275](https://arxiv.org/abs/2302.00275) find
  14. Lukas Haas, Michal Skreta, Silas Alberti, and Chelsea Finn. 2023b. Pigeon: Predicting image geolocations. Preprint, arXiv:2307.05845. — [arXiv:2307.05845](https://arxiv.org/abs/2307.05845) find
  15. James Hays and Alexei A. Efros. 2008. Im2gps: estimating geographic information from a single image. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. find
  16. James Hays and Alexei A. Efros. 2015. Large-scale image geolocalization. In Jaeyoung Choi and Gerald Friedland, editors, Multimodal Location Estimation of Videos and Images, pages 41–62. Springer International Publishing, Cham. find
  17. Martha Larson, Mohammad Soleymani, Guillaume Gravier, Bogdan Ionescu, and Gareth JF Jones. 2017. The benchmarking initiative for multimedia evaluation: Mediaeval 2016. IEEE MultiMedia, 24(1):93– 96. find
  18. Ling Li, Yu Ye, Bingchuan Jiang, and Wei Zeng. 2024. Georeasoner: Geo-localization with reasoning in street views using a large vision-language model. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  19. Ling Li, Yao Zhou, Yuxuan Liang, Fugee Tsung, and Jiaheng Wei. 2025. Recognition through reasoning: Reinforcing image geo-localization with large vision-language models. In Advances in Neural Information Processing Systems. find
  20. Grace Luo, Giscard Biamby, Trevor Darrell, Daniel Fried, and Anna Rohrbach. 2022. G3: Geolocation via guidebook grounding. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5841–5853, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  21. Ethan Mendes, Yang Chen, James Hays, Sauvik Das, Wei Xu, and Alan Ritter. 2024. Granular privacy control for geolocation with vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17243–17260, Miami, Florida, USA. Association for Com… find
  22. Eric Müller-Budack, Kader Pustu-Iren, and Ralph Ewerth. 2018. Geolocation estimation of photos using a hierarchical model and scene classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 563–579. Commonly referred to as ISNs (Individual Scene Networks). find
  23. OpenAI. 2025a. Chatgpt-4.1: Specialized coding and instruction-following model. Version released April 14, 2025. Accessed: April 20, 2026. find
  24. OpenAI. 2025b. Chatgpt-5: The fifth generation of generative pre-trained transformers. Version released August 7, 2025. Accessed: April 20, 2026. find
  25. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  26. Tuxun. 2022. Tuxun.fun. https://tuxun.fun/. Accessed: April 20, 2026. ↗
  27. Vercel. Vercel map generator. Accessed: 2025-12-24. find
  28. Vicente Vivanco Cepeda, Gaurav Kumar Nayak, and Mubarak Shah. 2023. GeoCLIP: Clip-inspired alignment between locations and images for effective worldwide geo-localization. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 18430–18445. find
  29. Nam Vo, Nathan Jacobs, and James Hays. 2017. Revisiting IM2GPS in the deep learning era. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2621–2630. find
  30. Chun Wang, Xiaojun Ye, Xiaoran Pan, Zihao Pan, Haofan Wang, and Yiren Song. 2025. Gre suite: Geolocalization inference via fine-tuned vision-language models and enhanced reasoning chains. Preprint, arXiv:2505.18700. — [arXiv:2505.18700](https://arxiv.org/abs/2505.18700) find
  31. P Jonathon Phillips, Amy N Yates, Ying Hu, Carina A Hahn, Eyal Norell, Alice J O’Toole, and 1 others. 2018. Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms. Proceedings of the National Academy of Sciences, 115(24):6171–6176. find
  32. Bradford Washburn. 1956. The camera eye vs. dr. cook. Life, 41(8):86–92. find
  33. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machin…
  34. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  35. N. Reimers and I. Gurevych. 2021. all-minilm-l6-v2 sentence transformer. Accessed: 2025-12-24. find
  36. Paul Hongsuck Seo, Tobias Weyand, and Bohyung Han. 2018. CPlaNet: Enhancing image geolocalization by combinatorial partitioning of maps. In Proceedings of the European Conference on Computer Vision (ECCV), pages 536–551. find
  37. Per-Kristian Helland slashP. 2023. Vali - create geoguessr locations like a pro. Accessed: 2025-12-24. find
  38. Zirui Song, Jingpu Yang, Yuan Huang, Jonathan Tonglet, Zeyu Zhang, Tao Cheng, Meng Fang, Iryna Gurevych, and Xiuying Chen. 2026. Geolocation with real human gameplay data: A large-scale dataset and human-like reasoning framework. Preprint, arXiv:2502.13759. — [arXiv:2502.13759](https://arxiv.org/abs/2502.13759) find
  39. Tobias Weyand, Ilya Kostrikov, and James Philbin. 2016. PlaNet - Photo Geolocation with Convolutional Neural Networks, page 37–55. Springer International Publishing. find
  40. Sahiti Yerramilli, Nilay Pande, Rynaa Grover, and Jayant Sravan Tamarapalli. 2025. GeoChain: Multimodal chain-of-thought for geographic reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23624–23639, Suzhou, China. Association for Computational Linguistics. find
  41. Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp. 2024. Can vision-language models be a good guesser? exploring vlms for times and location reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 636–645. find
  42. Zheyuan Zhang, Runze Li, Tasnim Kabir, and Jordan Lee Boyd-Graber. 2025. Navig: Natural language-guided analysis with vision language models for image geo-localization. ArXiv, abs/2502.14638. — [arXiv:2502.14638](https://arxiv.org/abs/2502.14638) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talreja-2026-georc-benchmark-geolocation-reasoning-chains/