Elamur Research⌕
☰
home / papers / 2026 / CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension

CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension

Paper · 2026 · Jingjie Zeng, Huayang Li, Liang Yang et al.

Summary

Word meanings tend to grow in a set direction: a concrete, embodied core sense gives rise to abstract senses through metaphor and metonymy (bread crust leads to Earth's crust). Benchmarks such as WSD and VQA check whether models recognize senses, but not whether they understand the direction of derivation. The authors built CogEvolve, an English benchmark that stores each word's senses as a directed acyclic graph across three aligned layers:

  • Symbolic: textual definitions.
  • Structural: derivation graphs.
  • Schematic: code-based SVG schemas that serve as a visual chain-of-thought.

LLMs tidied the candidate senses, and trained human annotators fixed the prototypes and derivation paths. Evaluation is tiered, moving from static recognition to reconstructing extension paths. In text, frontier models order senses nearly as well as humans, yet they often mistake the most frequent sense for the most basic one. With SVG schemas, their relational reasoning collapses even though recognition stays strong. The authors call this the Ungrounded Arrow: current architectures associate concepts well but lack grounded causal reasoning.

Key points

  • The candidate pool of 6,431 polysemous words came from WordNet, with relational data from ConceptNet. A human-architected, AI-assisted protocol was used, in which LLMs only consolidated and formatted definitions.
  • The three layers are L1 Symbolic, L2 Structural and L3 Schematic (SVG). The three evaluation tiers cover concept recognition (Tiers 1–2) and derivation-path reasoning (Tier 3).
  • Expert inter-annotator agreement was κ = 0.82. Human baselines were a naive group (N = 30) and an expert group (N = 10).
  • The authors name two failure patterns. The first is Frequency-Primacy Conflation, where statistical prevalence is confused with cognitive basicness in text. The second is the visual "Ungrounded Arrow" deficit, which leads them to describe models as "Super-Associators".
  • The benchmark is English-only and its SVGs are minimalist. Code and data are on GitHub (jjtail/CogEvolve-ACL2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dalian University of TechnologyDalian · China20 works · 46 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksEvaluation & benchmarks · 1,341 worksVision & language · 811 works

Cite

Cite as Jingjie Zeng, Huayang Li, Liang Yang, Yuanyuan Sun, Shaowu Zhang and Hongfei Lin. 2026. CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1190

@inproceedings{zeng2026cogevolve,
  title = {{CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension}},
  author = {Jingjie Zeng, Huayang Li, Liang Yang, Yuanyuan Sun, Shaowu Zhang and Hongfei Lin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1190},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1190}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing claude 4. https://www.anthropic.com/news/claude-4. ↗
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. find
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  4. Antonio Barcelona. 2012. Metaphor and metonymy at the crossroads: A cognitive perspective, volume 30. Walter de Gruyter. find
  5. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  6. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In International joint conference on artificial intelligence, pages 4330–4338. International Joint Conference on Artificial Intelligence, Inc. find
  7. Joanne Boisson, Asahi Ushio, Hsuvas Borkakoty, Kiamehr Rezaee, Dimosthenis Antypas, Zara Siddique, Nina White, and Jose Camacho-Collados. 2024. How are metaphors processed by language models? the case of analogies. In Proceedings of the 28th Conference on Computational Natural Language Learning, pag… find
  8. Gemini. 2025. A new era of intelligence with gemini 3. https://blog.google/products/gemini/ gemini-3/\#note-from-ceo. ↗
  9. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):155– 170.
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  11. Mathew Hardy, Ilia Sucholutsky, Bill Thompson, and Tom Griffiths. 2023. Large language models meet cognitive science: Llms as tools, models, and participants. In Proceedings of the annual meeting of the cognitive science society, volume 45. find
  12. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2025. Moral-bench: Moral evaluation of llms. ACM SIGKDD Explorations Newsletter, 27(1):62–71. find
  13. Mark Johnson. 2013. The body in the mind: The bodily basis of meaning, imagination, and reason. University of Chicago press. find
  14. Casey Kennington and Osama Natouf. 2022. The symbol grounding problem re-framed as concreteness-abstractness learned through spoken interaction. In Proceedings of the 26th Workshop on the Semantics and Pragmatics of Dialogue-Full Papers. find
  15. Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 4:169.
  16. George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press. find
  17. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.
  18. Fangyu Liu, Julian Eisenschlos, Jeremy Cole, and Nigel Collier. 2022. Do ever larger octopi still amplify reporting biases? evidence from judgments of typical colour. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Intern… find
  19. Meta. 2024. Introducing llama 3.1: Our most capable models to date. https://ai.meta.com/blog/ meta-llama-3-1. ↗
  20. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  21. Eleanor Rosch. 1975. Cognitive representations of semantic categories. Journal of experimental psychology: General, 104(3):192. find
  22. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. find
  23. James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, and 1 others. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7):1285–1295. find
  24. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  25. Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. Metaphor understanding challenge dataset for llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3517–3536. xAI. 2025. Grok 4 fast pushing the frontie…
  26. Huatao Xu, Liying Han, Qirui Yang, Mo Li, and Mani Srivastava. 2024. Penetrative ai: Making llms comprehend the physical world. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications, pages 1–7. find
  27. Mo Yu, Lemao Liu, Junjie Wu, Tsz Ting Chung, Shunchi Zhang, Jiangnan Li, Dit-Yan Yeung, and Jie Zhou. 2025. The stochastic parrot on llm’s shoulder: A summative assessment of physical concept understanding. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associati… find
  28. Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. 2024. Vgbench: A comprehensive benchmark of vector graphics understanding and generation for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3647–3659. find
  29. Mistral. 2024. Introducing the world’s best edge models. https://mistral.ai/news/ministraux/. ↗
  30. OpenAI. 2024. Hello gpt-4o. https://openai.com/ index/hello-gpt-4o. ↗
  31. OpenAI. 2025. Gpt-5.1: A smarter, more conversational chatgpt. https://openai.com/index/gpt-5-1/. ↗
  32. Team Qwen. 2025. Qwen3: Think deeper, act faster. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-cogevolve-multimodal-benchmark-evaluating-relational/