Elamur Research⌕
☰
home / papers / 2026 / Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning

Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning

Summary

Vision–language models often fail at abstract visual puzzles such as Bongard problems, and it is unclear whether the weakness lies in reasoning or in how images are encoded. The authors test this on Bongard-LOGO, a synthetic benchmark whose images are generated from known LOGO drawing programs. Their Componential–Grammatical (C–G) setup removes pixels from the pipeline. Each example reaches a language model either as its ground-truth action program or as a natural-language rendering of that program. The model must then infer the hidden rule and label a query example. Across 12 language models, this symbolic framing clearly outperforms a strong image-based baseline, Gemini-2.5-Flash, which stays near chance. Changing the text format, adding concept labels or adding a single visual anchor matters much less than moving from pixels to explicit structure. The authors present C–G as a diagnostic upper bound rather than a practical system. They conclude that representation is a central bottleneck, which supports modular designs where a front-end produces a structured visual language for an LLM to reason over.

Key points

  • Data: Bongard-LOGO (12,000 problems), where each problem has six positive and six negative examples plus a query. Results are reported on the Free-form (FF), Basic (BD), HD-Comb and HD-Novel splits.
  • Two symbolic interfaces are compared: Action Program (AP) grammar strings built from LOGO primitives (lines, arcs, turn angles) and Action Description (AD) text derived from them.
  • The visual baseline Gemini-2.5-Flash scores about 50%. Symbolic input gives roughly 25–30 point gains, and Phi-4-Reasoning reaches 96.2% on Free-form problems.
  • Ablations (concept prompts, minimal-context prompts, grounded variants, token randomization) produce smaller, model-dependent effects than the switch to symbolic input.
  • Limitations: the setup assumes oracle programs, so perception is bypassed. Human-designed problems remain hard even with perfect input, and transfer to natural-image sets such as Bongard-OpenWorld or Bongard-HOI is untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.19

@inproceedings{vaishnav2026symbolic,
  title = {{Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning}},
  author = {Vaishnav, Mohit and Tammet, Tanel},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.19},
  url = {https://doi.org/10.18653/v1/2026.conll-main.19}
}

References

42 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paweł Batorski, Jannik Brinkmann, and Paul Swoboda. 2025. Nsa: Neuro-symbolic arc challenge. arXiv preprint arXiv:2501.04424. — [arXiv:2501.04424](https://arxiv.org/abs/2501.04424) find
  2. Mikhail M Bongard. 1970. Pattern Recognition. Hayden Book Company, Rochelle Park, NJ. find
  3. Declan Campbell, Sunayana Rane, Tyler Giallanza, C Nicolò De Sabbata, Guillermo Ortiz-Jiménez, Pascal Frossard, and Olga Russakovsky. 2024. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37. find
  4. Stefan Depeweg, Contantin A Rothkopf, and Frank Jäkel. 2024. Solving bongard problems with a visual language and pragmatic constraints. Cognitive Science, 48(5):e13432. find
  5. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20406–20417. find
  6. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR. find our card
  7. Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. 2022. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19735–19745. find
  8. Idan Fine and Omri Abend. 2025. A fine-grained analysis of multi-object representation in clip encoders. arXiv preprint arXiv:2502.19842. Submitted to ICLR 2025. — [arXiv:2502.19842](https://arxiv.org/abs/2502.19842) find
  9. Cheongwoong Kang and Jaesik Choi. 2023. Impact of co-occurrence on factual knowledge of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7770–7791. find
  10. Harry E Foundalis. 2006. Phaeaco: A cognitive architecture inspired by bongard’s problems. Doctoral Dissertation, Indiana University. find
  11. Haoqiang Kang, Enna Sachdeva, Piyush Gupta, Sangjae Bae, and Kwonjoon Lee. 2025. Gflowvlm: Enhancing multi-step reasoning in vision-language models with generative flow networks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3815–3825. find
  12. Dedre Gentner and Keith J Holyoak. 1997. Reasoning and learning by analogy. American Psychologist, 52(1):32–34. find
  13. Dedre Gentner and Christian Hoyos. 2017. Analogy and abstraction. Topics in Cognitive Science, 9(3):672– 693. find
  14. James Gips. 1975. Shape grammars and their uses: Artificial perception, shape generation and computer aesthetics. PhD thesis, Stanford University. find
  15. Google DeepMind. 2024. Gemini 2.0 flash: Technical report. https://ai.google.dev/gemini-api/ docs. Accessed October 2025. ↗
  16. Tetiana Grinberg, Dmitry Grinberg, Grigory Shcherban-skiy, and Lev Cherbanski. 2023. Bongard architecture: Towards scalable and transparent machine reasoning. AAAI Spring Symposium Series EDGeS Submission. find
  17. Daya Guo and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. find our card
  18. Douglas R Hofstadter. 1979. Gödel, Escher, Bach: An Eternal Golden Braid. Basic Books. find
  19. Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2023. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In Advances in Neural Information Processing Systems, volume 36, pages 38957–38977. find
  20. Terry Knight and George Stiny. 2015. Making grammars: From computing with shapes to computing with things. Design Studies, 41:8–28. find
  21. Peter Kulits, Haiwen Feng, Weiyang Liu, Victoria Abre-vaya, and Michael J Black. 2024. Re-thinking inverse graphics with large language models. arXiv preprint arXiv:2404.15228. — [arXiv:2404.15228](https://arxiv.org/abs/2404.15228) find
  22. Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. 2015. Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems, volume 28. find
  23. Quang-Hung Le, Long Hoang Dang, Ngan Hoang Le, Truyen Tran, and Thao Minh Le. 2025. Progressive multi-granular alignments for grounded reasoning in large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4473–4481. find
  24. Haoxin Li, Bohao Liu, Hanyang Zhang, and Heng Huang. 2025. Enhancing vision-language compositional understanding via synthetic perturbations. arXiv preprint arXiv:2503.10825. — [arXiv:2503.10825](https://arxiv.org/abs/2503.10825) find
  25. Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2023. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910–10921. find
  26. Microsoft Research. 2024. Introducing phi-4: Microsoft’s newest small language model specializing in complex reasoning. https://techcommunity.microsoft.com/blog/ azure-ai-foundry-blog/introducing-phi-4. ↗
  27. Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. 2020. Bongard-logo: A new benchmark for human-level concept learning and reasoning. In Advances in Neural Information Processing Systems, volume 33, pages 16468–16480. find
  28. Fiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto, Lorenzo Baraldi, and Rita Cucchiara. 2025. Causal graphical models for vision-language compositional understanding. In International Conference on Learning Representations (ICLR). find
  29. Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. 2024. Synthesize, diagnose, and optimize: Towards fine-grained vision-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26161–26171. find
  30. Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. 2024. Prism: A framework for decoupling and assessing the capabilities of vlms. In Advances in Neural Information Processing Systems. find
  31. Bongard-openworld: Few-shot reasoning for free-form visual concepts in the real world. In International Conference on Learning Representations (ICLR). find
  32. Antje Wüst, Sven Magg, and Stefan Wermter. 2024. Bongard in wonderland: Visual puzzles that still make ai models hallucinate. arXiv preprint arXiv:2410.19546. — [arXiv:2410.19546](https://arxiv.org/abs/2410.19546) find
  33. Yichang Xu, Gaowen Liu, Ramana Rao Kompella, Sihao Hu, Tiansheng Huang, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, and Ling Liu. 2025. Language-vision planner and executor for text-to-visual reasoning. arXiv preprint arXiv:2506.07778. — [arXiv:2506.07778](https://arxiv.org/abs/2506.07778) find
  34. An Yang and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. find our card
  35. Ilker Yildirim, Mario Belledonne, Winrich Freiwald, and Josh Tenenbaum. 2020. Efficient inverse graphics in biological face processing. Science Advances, 6(10):eaax5979. find
  36. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936. — [arXiv:2210.01936](https://arxiv.org/abs/2210.01936) find
  37. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8… find our card
  38. Rahul Ramachandran, Ali Garjani, Roman Bachmann, Andrei Atanov, Oğuzhan Fatih Kar, and Amir Zamir. 2025. How well does gpt-4o understand vision? evaluating multimodal foundation models on standard computer vision tasks. arXiv preprint arXiv:2507.01955. — [arXiv:2507.01955](https://arxiv.org/abs/2507.01955) find
  39. Ruizhuo Song and Beiming Yuan. 2024. Solving the clustering reasoning problems by modeling a deep-learning-based probabilistic model. arXiv preprint arXiv:2403.03173. — [arXiv:2403.03173](https://arxiv.org/abs/2403.03173) find
  40. George Stiny and James Gips. 1972. Shape grammars and the generative specification of painting and sculpture. In Information Processing 71: Proceedings of the IFIP Congress 1971, volume 2, pages 1460–1465. North-Holland. find
  41. Mohit Vaishnav and Tanel Tammet. 2025. A cognitive paradigm approach to probe the perception-reasoning interface in vlms. arXiv preprint arXiv:2501.13620. — [arXiv:2501.13620](https://arxiv.org/abs/2501.13620) find
  42. Rujie Wu, Xiaojian Ma, Zhenliang Zhang, Wei Wang, Qing Li, Song-Chun Zhu, and Yizhou Wang. 2024. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vaishnav-2026-symbolic-grounding-reveals-representational-bottlenecks/