📄 SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data

Summary

Vision-language models (VLMs) handle captioning and VQA well, but they remain weak at judging how objects are arranged relative to one another. The authors trace part of this weakness to training data. Across popular VL corpora, spatial relations are scarce: a handful such as "left" or "on" dominate, and most others sit in a long tail. To address this, they convert hyper-detailed image captions from DOCCI, Localized Narratives and PixMo-Cap into a synthetic spatial VQA corpus. Qwen2.5-3B-Instruct is prompted to derive question-answer pairs from each description, and the original real photographs are kept as the images. Fine-tuning Qwen2VL models on this data produces the SpaRE family. These models score higher on VSR, What's Up, 3DSRBench and RealWorldQA, with the largest jump reaching 49% on one What's Up split. They also keep their results on general benchmarks like MMMU and MMBench. The study shows that cheap, LLM-generated supervision grounded in real images can close a skill gap relevant to robotics, navigation and assistive tools, without expensive manual annotation.

Key points

• Data imbalance: In widely used VL training sets, the top 17% of spatial relations cover roughly 90% of all spatial examples. Relations such as facing, opposite or surrounding are barely represented.
• Synthetic corpus: After filtering, the dataset contains about 455k images with roughly 3.4M QA pairs. PixMo-Cap contributes about 2.0M pairs, Localized Narratives about 1.2M, and DOCCI about 108k.
• Spatial benchmark gains: SpaRE models improve average spatial-benchmark accuracy by about 9% (2B) and 3% (7B). On RealWorldQA, the gains are 4.9% and 1.1% respectively.
• Comparison with caption training: Fine-tuning on targeted spatial QA beats training directly on captions. Average spatial accuracy is 82.3% for SpaRE-7B, 79.2% for Qwen2VL-7B and 76.6% for Molmo-7B-D.
• Remaining weakness: Perspective-taking is still hard, which shows up as weaker results on 3DSRBench. The authors attribute this to an egocentric bias inherited from the source captions. Off-topic but image-relevant generated pairs appear to help preserve general performance.