Elamur Research⌕
☰
home / papers / 2025 / CountQA: How Well Do MLLMs Count in the Wild?

CountQA: How Well Do MLLMs Count in the Wild?

Paper · 2025 · Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande et al.

Summary

Multimodal LLMs describe images fluently, but they are unreliable at counting objects, especially in crowded everyday scenes. Existing benchmarks do not test this well. Crowd-counting datasets cover only people, and general MLLM or VQA suites include few hard enumeration questions. To fill this gap, the authors built CountQA, a benchmark of 1,528 question-answer pairs. All images were photographed by the authors themselves and show cluttered, dense, partly occluded indoor and outdoor scenes. The true count was established on site, so annotators could move around the scene and recount instead of relying on a single photo. They tested 15 proprietary and open-source MLLMs zero-shot, requiring a single integer answer. The best model reached only 42.9% Exact Match, and accuracy dropped as the number of objects grew. The authors attribute this mainly to perception rather than arithmetic. They point to two causes: vision encoders trained for holistic image-text alignment, and low fixed input resolutions. This makes CountQA a useful diagnostic for building MLLMs that are better grounded in space and number.

Key points

  • Dataset: CountQA has 1,528 QA pairs, released on Hugging Face.
    • Questions were drafted with Gemini 2.5 Flash from the on-site counts.
    • They include simple questions ("How many X?") and combined ones ("How many X and Y?").
  • Annotations: each image has a list of countable object types, one of 10 scene categories, and an is_focused tag that separates clean shots from busy, cluttered ones.
  • Evaluation setup: 15 MLLMs, both API-based and open-source, were tested zero-shot.
    • Models included Gemini 2.5 Pro, OpenAI o3, Claude 4 Sonnet, Pixtral Large, Gemma3 and Llava.
    • Temperature was set to 0, and Exact Match was the primary metric.
  • Results: the top model scored 42.9% EM and the runner-up 34.42%. Several open-source models scored below 15%.
  • Limitations and future work:
    • The images may carry a regional bias, and the dataset is small because of labor-intensive annotation.
    • Comparison and parity questions are not covered.
    • The authors propose better vision-language fusion, perception-aware training objectives, and MLLMs that call tools such as SAM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksComputer Vision and Pattern Recognition · 3,621 works

Cite

Cite as Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande and Sahiti Yerramilli. 2025. CountQA: How Well Do MLLMs Count in the Wild?. arXiv:2508.06585.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.06585 · arXiv 2508.06585

@misc{tamarapalli2025countqa,
  title = {{CountQA: How Well Do MLLMs Count in the Wild?}},
  author = {Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande and Sahiti Yerramilli},
  year = {2025},
  doi = {10.48550/arxiv.2508.06585},
  eprint = {2508.06585},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.06585}
}

References

47 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas,… — Pixtral 12B
  2. Niki Amini-Naieni, Tengda Han, and Andrew Zisserman. Countgd: Multi-modal open-world counting, 2025. 2 find
  3. Anthropic. Introducing claude 4, 2025. Accessed: 2025-06-28. 7 find
  4. Anthropic. Claude 3.5 sonnet, 2025. Accessed May 18, 2025. 1 find
  5. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Ha… — Qwen2.5-VL Technical Report
  6. Feng Chen, Chenhui Gou, Jing Liu, Yang Yang, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, and Qi Wu. Evaluating and advancing multimodal large language models in perception ability lens, 2025. 3 find
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, Krishna Harida… — Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
  8. Gemini-Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… — Gemini: A Family of Highly Capable Multimodal Models
  9. Gemma-Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  10. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017. 3
  11. Rynaa Grover, Jayant Sravan Tamarapalli, Sahiti Yerramilli, and Nilay Pande. Huemanity: Probing fine-grained visual perception in mllms. arXiv preprint arXiv:2506.03194, 2025. 1 — [arXiv:2506.03194](https://arxiv.org/abs/2506.03194) find
  12. Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 3
  13. Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Rajpoot, and Mubarak Shah. Composition loss for counting, density map estimation and localization in dense crowds, 2018. 3 find
  14. Aishwarya Jadhav, Jeffery Cao, Abhishree Shetty, Urvashi Kumar, Aditi Sharma, Ben Sukboontip, Jayant Tamarapalli, Jingyi Zhang, and Aniruddh Koul. Ai guide dog: Egocentric path prediction on smartphone. Proceedings of the AAAI Symposium Series, 5(1):220–227, 2025. find
  15. Vidhi Jain, Jayant Sravan Tamarapalli, Sahiti Yerramilli, and Yonatan Bisk. Maea: Multimodal attribution for embodied ai. arXiv preprint arXiv:2307.13850, 2023. — [arXiv:2307.13850](https://arxiv.org/abs/2307.13850) find
  16. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  17. Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei-Lun Chao. Mllm-compbench: A comparative reasoning benchmark for multimodal llms, 2025. 3 find
  18. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. 2
  19. Victor Lempitsky and Andrew Zisserman. Learning to count objects in images. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2010. 2 find
  20. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024. 3 find
  21. Jiansheng Li, Xingxuan Zhang, Hao Zou, Yige Guo, Renzhe Xu, Yilong Liu, Chuzhao Zhu, Yue He, and Peng Cui. Counts: Benchmarking object detectors and multimodal large language models under distribution shifts, 2025. 1, 3 find
  22. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1, 6, 7
  23. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024. 2
  24. Mistral AI. Announcing pixtral large, 2024. Accessed: 2025-06-28. 7 find
  25. Mistral AI. Introducing mistral medium 3, 2025. Accessed: 2025-06-28. 7 find
  26. Mistral AI. Announcing mistral small 3.1, 2025. Accessed: 2025-06-28. 7 find
  27. OpenAI. Introducing openai o3 and o4-mini, 2025. Accessed: 2025-06-28. 7 find
  28. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — GPT-4 Technical Report
  29. Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Mikel Artetxe, and Yi T… — Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models
  30. Nilay Pande and Suyash P. Awate. Generative deep-neural-network mixture modeling with semi-supervised minmax+em learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 5666–5673, 2021. find
  31. Muhammad Fetrat Qharabagh, Mohammadreza Ghofrani, and Kimon Fountoulakis. Lvlm-count: Enhancing the counting ability of large vision-language models, 2025. 1, 5, 6, 8 find
  32. Yifei Qian, Zhongliang Guo, Bowen Deng, Chun Tong Lei, Shuai Zhao, Chun Pong Lau, Xiaopeng Hong, and Michael P. Pound. T2icount: Enhancing cross-modal understanding for zero-shot counting, 2025. 3 find
  33. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2, 7
  34. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment an… find
  35. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection, 2016. 2 find
  36. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016. 2
  37. Zenglin Shi, Ying Sun, and Mengmi Zhang. Training-free object counting with prompts, 2023. 3, 6 find
  38. Vishwanath A. Sindagi, Rajeev Yasarla, and Vishal M. Patel. Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method, 2020. 3 find
  39. Yanpeng Sun, Shan Zhang, Wei Tang, Aotian Chen, Piotr Koniusz, Kai Zou, Yuan Xue, and Anton van den Hengel. Mathglance: Multimodal large language models do not know where to look in mathematical diagrams, 2025. 1 find
  40. Carnegie Mellon University. Embodied symbiotic assistants that see, act, infer and chat. 2023. find
  41. Gaurav Verma, Rynaa Grover, Jiawei Zhou, Binny Mathew, Jordan Kraemer, Munmun De Choudhury, and Srijan Kumar. A community-centric perspective for characterizing and detecting anti-asian violence-provoking speech, 2024. find
  42. Qi Wang, Junyu Gao, Wei Lin, and Xuelong Li. Nwpu-crowd: A large-scale benchmark for crowd counting and localization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):2141–2149, 2021. 3 find
  43. Sahiti Yerramilli, Jayant Sravan Tamarapalli, Jonathan Francis, and Eric Nyberg. Attribution regularization for multimodal paradigms. arXiv preprint arXiv:2404.02359, 2024. — [arXiv:2404.02359](https://arxiv.org/abs/2404.02359) find
  44. Sahiti Yerramilli, Jayant Sravan Tamarapalli, Tanmay Girish Kulkarni, Jonathan Francis, and Eric Nyberg. Semantic augmentation in images using language. arXiv preprint arXiv:2404.02353, 2024. — [arXiv:2404.02353](https://arxiv.org/abs/2404.02353) find
  45. Sahiti Yerramilli, Nilay Pande, Rynaa Grover, and Jayant Sravan Tamarapalli. Geochain: Multimodal chain-of-thought for geographic reasoning. arXiv preprint arXiv:2506.00785, 2025. 1 find
  46. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023. 7
  47. Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 589–597, 2016. 2, 3 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tamarapalli-2025-countqa-well-mllms-count-wild/