Elamur Research⌕
☰
home / papers / 2026 / EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

Paper · 2026 · Jianfei Wu, Zhichun Wang, Zhensheng Wang et al.

Summary

Most geo-spatial question answering (GSQA) benchmarks test static lookups, such as the distance between two landmarks. Real trip planning is harder because it depends on where the user is now and often has several goals at once. The authors address this with EVGeoQA, a benchmark built around electric-vehicle charging. Each query is tied to the user's current coordinates and asks for a charging station that also has a desired activity nearby, such as a café, within walking distance. The data combine State Grid charging stations and nearby POIs from the Gaode API in three Chinese cities of different sizes. User locations are synthesized by K-Means clustering of Baidu heatmaps, weighting population density and roads. Queries come from templates. The authors also propose GeoRover, a tool-using agent framework for evaluating LLMs. Models handle individual sub-steps well, but accuracy falls as the target gets farther away. Models often stop searching too early. Some models, without being instructed to, reuse their earlier search path to explore more efficiently.

Key points

  • Dataset: EVGeoQA covers Hangzhou, Qingdao and Linyi.
    • Charging stations: 258, 165 and 157 respectively.
    • QA pairs: 19,940, 14,162 and 14,416, about 48.5k in total.
    • Each city has about 1,000 synthesized user locations.
  • Evaluation framework: GeoRover gives the agent four interactive geo-spatial tools, including ChangeLocation for moving the search area and CalculateDistance.
  • Distance effect: average Hits@2 drops from about 50% to about 38% as the search radius grows.
  • Model comparisons:
    • Thinking-mode models do better; in the Hangzhou "No Limit" setting, Qwen3-8B with thinking scores 0.3452 versus 0.2889 without it.
    • Larger models such as Qwen2.5-72B and Gemini-2.5-Pro lead across metrics.
  • Failure analysis:
    • Agents call ChangeLocation rarely; the average in Linyi is 0.58, 2.07 and 3.78 calls per episode for the <10km, <20km and No Limit tiers.
    • The main error types are insufficient exploration ("laziness"), mixing up facts in long contexts, wrong tool arguments, and hitting the tool-call limit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beijing Normal UniversityBeijing · China22 works · 3,177 citations🎓National University of Defense TechnologyChangsha · China20 works · 1,339 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksSignal Processing · 493 works

Cite

Cite as Jianfei Wu, Zhichun Wang, Zhensheng Wang and Zhiyu He. 2026. EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.186

@inproceedings{wu2026evgeoqa,
  title = {{EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration}},
  author = {Jianfei Wu, Zhichun Wang, Zhensheng Wang and Zhiyu He},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.186},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.186}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The EVGeoQA benchmark is constructed based on urban data from three distinct Chinese cities of varying scales, with QA pairs predominantly in Chinese. This linguistic specificity may introduce inherent biases. Our future work aims to mitigate this by incorporating multi-lingual data and a broader ra… find
  2. Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. 2020. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8):1295. find
  3. Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. 2022. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In NIPS, volume 33, pages 1877– 1901.
  5. Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. 2023. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05029. — [arXiv:2310.05029](https://arxiv.org/abs/2310.05029) find
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  7. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491– 6501.
  8. Jie Feng, Tianhui Liu, Yuwei Du, Siqi Guo, Yuming Lin, and Yong Li. 2025. Citygpt: Empowering urban spatial cognition of large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 591–602. find
  9. Yu Feng, Linfang Ding, and Guohui Xiao. 2023. Geoqamap-geographic question answering with maps leveraging llm and open knowledge base. In 12th International Conference on Geographic Information Science (GIScience 2023), volume 277, page 28. find
  10. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964. find
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  12. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–291…
  13. Sergios-Anestis Kefalidis, Dharmen Punjani, Konstantinos Plasa Eleni Tsalapati, Mariangela Pollali, Myrto Tsokanaridou Michail Mitsios, Manolis Koubarakis, and Pierre Maret. 2023. Benchmarking geospatial question answering engines using the dataset geoquestions1089. In The Semantic Web -ISWC 2023 -… find
  14. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  15. Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context llms struggle with long in-context learning. Transactions on Machine Learning Research.
  16. Zekun Li, Malcolm Grossman, Mihir Kulkarni, Muhao Chen, Yao-Yi Chiang, and 1 others. 2025. Mapqa: Open-domain geospatial question answering on map data. arXiv preprint arXiv:2503.07871. — [arXiv:2503.07871](https://arxiv.org/abs/2503.07871) find
  17. Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. 2023. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. arXiv preprint arXiv:2304.13343, 10. — [arXiv:2304.13343](https://arxiv.org/abs/2304.13343) find
  18. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  19. Tong Lu, Zhichun Wang, Yaoyu Zhou, Yiming Guan, Zhiyong Bai, and Junsheng Du. 2026. Scimkg: A multimodal knowledge graph for science education with text, image, video and audio. 40(18):15466– 15474. find
  20. Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. 2022. Sqa3d: Situated question answering in 3d scenes. In The Eleventh International Conference on Learning Representations. find
  21. Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, Chris Cundy, Ziyuan Li, Rui Zhu, and Ni Lao. 2023. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.0… find
  22. Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, and 1 others. 2024. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF conference on computer vi… find
  23. OpenAI. 2025. gpt-oss-120b and gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  24. Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357–2368.
  25. Ralf Philipsen, Teresa Brell, Waldemar Brost, Teresa Eickels, and Martina Ziefle. 2018. Running on empty – users’ charging behavior of electric vehicles versus traditional refueling. Transportation Research Part F: Traffic Psychology and Behaviour. find
  26. Dharmen Punjani, K. Singh, Andreas Both, Manolis Koubarakis, I. Angelidis, Konstantina Bereta, Themis Beris, Dimitris Bilidas, Theofilos Ioannidis, Nikolaos Karalis, Christoph Lange, D. Pantazi, Christos Papaloukas, and Georgios I. Stamoulis. 2018. Template-based question answering over linked geosp… find
  27. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, and 1 others. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations.
  28. QwenTeam. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  29. QwenTeam. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  30. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551.
  31. Kuldeep Singh, Simerjot Kaur, and Charese Smiley. 2024. Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline. In Proceedings of the 5th ACM International Conference on AI in Finance, pages 266–273. find
  32. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2998–3009.
  33. Jianing Sun, Zhichao Zhang, and Xueli He. 2024. Llm4edukg: Llm for automatic construction of educational knowledge graph. In 2024 International Conference on Networking and Network Applications (NaNA), pages 269–275. IEEE. find
  34. Jingru Wang, Wen Ding, and Xiaotong Zhu. 2025a. Financial analysis: Intelligent financial data analysis system based on llm-rag. arXiv preprint arXiv:2504.06279. — [arXiv:2504.06279](https://arxiv.org/abs/2504.06279) find
  35. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the 2024 CHI conference on human factors in computing systems, pages 1–21. find
  36. Yu Zheng, Licia Capra, Ouri Wolfson, and Hai Yang. 2014. Urban computing: Concepts, methodologies, and applications. ACM Trans. Intell. Syst. Technol., 5:38:1–38:55. find
  37. Gengze Zhou, Yicong Hong, and Qi Wu. 2024. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649. find
  38. Zhensheng Wang, Wenmian Yang, Kun Zhou, Yiquan Zhang, and Weijia Jia. 2025b. Retqa: A large-scale open-domain tabular question answering dataset for real estate sector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25452–25460. find
  39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  40. Jianfei Wu, Xubin Wang, and Weijia Jia. 2025. Enhancing text annotation through rationale-driven collaborative few-shot prompting. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  41. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, and 1 others. 2025. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101.
  42. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: a benchmark for real-world planning with language agents. In Proceedings of the 41st International Conference on Machine Learning, pages 54590–54613.
  43. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  44. Dazhou Yu, Riyang Bao, Gengchen Mai, and Liang Zhao. 2025. Spatial-rag: Spatial retrieval augmented generation for real-world spatial reasoning questions. arXiv e-prints, pages arXiv–2502. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-evgeoqa-benchmarking-llms-dynamic-multi/