Elamur Research⌕
☰
home / papers / 2026 / Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

Paper · 2026 · Yichi Zhang, Zhuo Chen, Lingbing Guo et al.

Summary

Multi-modal LLMs handle natural photos well, but they struggle with images whose meaning comes from human-designed abstractions. This paper looks at one such case that has received little study: images of multi-modal relational knowledge (MMRK). These are node-edge graphs whose entities carry pictures and text. The authors call reasoning over them STAR (STructured and Abstractive Reasoning). They identify two gaps: no scalable source of training data, and no systematic recipe for training or evaluation.

To fill the data gap, they built an automatic engine. It samples subgraphs from three public multi-modal knowledge graphs, renders them as images, and pairs each image with an instruction and a chain-of-thought rationale. This produces the STAR-64K dataset. Training then runs in two stages:

  1. supervised fine-tuning;
  2. targeted optimization, using either preference alignment or KGRPO, a GRPO variant with an added reward for factually correct steps inside the reasoning chain.

After this training, small 3B/7B models beat GPT-4o on STAR tasks and keep their general knowledge. This suggests that synthetic data built from knowledge graphs is an inexpensive way to teach structured visual reasoning.

Key points

  • Data sources and dataset. Subgraphs come from VisualSem, FB15K-237 and MKG-Y. Sampling starts from one entity and continues by random walk. The resulting STAR-64K contains about 64K instruction samples with CoT annotations.
  • Eight seed tasks in two groups.
    • Understanding: Entity, Relation, Image and Triple Counting, plus Subgraph Description.
    • Reasoning: Error Detection, Entity Reasoning and Relation Reasoning, modelled on knowledge graph completion.
  • Two-stage training.
    • Stage 1 is SFT.
    • Stage 2 uses either DPO/ORPO/SimPO or GRPO/KGRPO. KGRPO's knowledge-informed reward is meant to reduce hallucinated facts in the CoT.
  • Models and evaluation. Backbones range from 3B to 34B, including Qwen2.5-VL (3B/7B/32B), LLaVA-1.5-7B and LLaVA-NEXT-34B. The paper gives the number of open-source MLLMs as 8 in one place and 5 in another. A dedicated evaluation protocol is defined for each task.
  • Ablations and retention.
    • Removing CoT from the training data lowers scores for every model tested.
    • Dropping entity images or entity texts both hurt performance, with texts mattering more.
    • On MMMU, two-stage training preserves and in some domains improves general knowledge. KGRPO retains more than DPO does.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 worksSummarization · 232 works

Cite

Cite as Yichi Zhang, Zhuo Chen, Lingbing Guo, Wen Zhang and Huajun Chen. 2026. Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.761

@inproceedings{zhang2026structured,
  title = {{Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images}},
  author = {Yichi Zhang, Zhuo Chen, Lingbing Guo, Wen Zhang and Huajun Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.761},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.761}
}

References

58 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Houda Alberts, Teresa Huang, Yash Deshpande, Yibo Liu, Kyunghyun Cho, Clara Vania, and Iacer Calixto. 2020. Visualsem: a high-quality knowledge graph for vision and language. CoRR, abs/2008.09150. find
  2. Shuowen An, Si Zhang, Tongyu Guo, Shuang Lu, Wenying Zhang, and Zhihui Cai. 2025. Impacts of generative AI on student teachers’ task performance and collaborative knowledge construction process in mind mapping-based collaborative environment. Comput. Educ., 227:105227. find
  3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. arXiv prepr…
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. arXiv preprint arXiv:2…
  5. Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD Conference, pages 1247–1250. ACM. find
  6. Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, Jiaqi Li, Xiaoze Liu, Jeff Z. Pan, Ningyu Zhang, and Huajun Chen. 2024. Knowledge graphs meet multi-modal learning: A comprehensive survey. CoRR, abs/2402.05391. — [arXiv:2402.05391](https://arxiv.org/abs/2402.05391) find
  7. Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, and 2 others. 2024. A survey on multimodal large language models for autonomou… find
  8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248– 255. IEEE Computer Society. find
  9. John Ellson, Emden R. Gansner, Eleftherios Koutsofios, Stephen C. North, and Gordon Woodhull. 2004. Graphviz and dynagraph - static and dynamic graph drawing tools. In Graph Drawing Software, pages 127–148. Springer. find
  10. Chaoyou Fu, Yifan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, Caifeng Shan, and Ran He. 2024a. Mme-survey: A comprehensive survey on evaluation of multimodal llms. CoRR, abs/2411.15296. — [arXiv:2411.15296](https://arxiv.org/abs/2411.15296) find
  11. Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, and 5 others. 2024b. Ocrbench v2: An improved benchmark for evaluating large multimodal… — [arXiv:2501.00321](https://arxiv.org/abs/2501.00321) find
  12. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models.…
  13. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through rein… find
  14. Jiwoo Hong, Noah Lee, and James Thorne. 2024. ORPO: monolithic preference optimization without reference model. In EMNLP, pages 11170–11189. Association for Computational Linguistics. find
  15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
  16. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Yuexian Zou, Zhou Zhao, and Shinji Watanabe. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In AAAI, pages 23802– 23804.… find
  17. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In ECCV (4), Lecture Notes in Computer Science, pages 235– 251. Springer.
  18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  19. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv: 2411.16594. find
  20. Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang, Lei Lu, Zhangdai-hong Liu, Yefeng Zheng, Xu Sun, Yang Yang, Lei A. Clifton, and David A. Clifton. 2023a. A medical multimodal large language model for future pandemics. npj Digit. Medicine, 6. find
  21. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. Improved baselines with visual instruction tuning.
  22. Ye Liu, Hui Li, Alberto García-Durán, Mathias Niepert, Daniel Oñoro-Rubio, and David S. Rosenblum. 2019. MMKG: multi-modal knowledge graphs. In ESWC, volume 11503 of Lecture Notes in Computer Science, pages 459–474. Springer. find
  23. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR (Poster). Open-Review.net. find
  24. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR. Open-Review.net.
  25. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS. find
  26. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. 2022a. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL (Findings), pages 2263–2279. Association for Computational Linguistics.
  27. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. 2022b. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL (Findings), Findings of ACL, pages 2263–2279. Association for Computational Linguistics.
  28. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In NeurIPS. find
  29. Roberto Navigli and Simone Paolo Ponzetto. 2010. Babelnet: Building a very large multilingual semantic network. In ACL, pages 216–225. The Association for Computer Linguistics.
  30. OpenAI. 2023. GPT-4 technical report. abs/2303.08774.
  31. CoRR, OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276.
  32. Preprint, Haojie Pan, Yuzhou Zhang, Zepeng Zhai, Ruiji Fu, Ming Liu, Yangqiu Song, Zhongyuan Wang, and Bing Qin. 2022. Kuaipedia: a large-scale multi-modal short-video encyclopedia. abs/2211.00732. — [arXiv:2211.00732](https://arxiv.org/abs/2211.00732) find
  33. CoRR, Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS.
  34. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256.
  35. Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326. find
  36. Shezheng Song, Xiaopeng Li, and Shasha Li. 2023. How to bridge the gap between modalities: A comprehensive survey on multimodal large language model. CoRR, abs/2311.07594. — [arXiv:2311.07594](https://arxiv.org/abs/2311.07594) find
  37. Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In WWW, pages 697–706. ACM. find
  38. Qwen Team. 2024. Qvq: To see the world with wisdom. find
  39. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pages 9568–9578. IEEE.
  40. Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM, 57(10):78–85.
  41. Xueyao Wan and Hang Yu. 2025. Mmgraphrag: Bridging vision and language with interpretable multimodal knowledge graphs. Preprint, arXiv:2507.20804. — [arXiv:2507.20804](https://arxiv.org/abs/2507.20804) find
  42. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. In NeurIPS. find
  43. Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. 2022. Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion. In ACM Multimedia, pages 3857–3866. ACM. find
  44. Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG-BERT: BERT for knowledge graph completion. CoRR, abs/1909.03193. find
  45. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024a. Mmmu: A massive multi-discipline multimodal…
  46. Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024b. MMMU: A massive multi-discipline multimodal…
  47. Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. 2025a. Videollama 3: Frontier multimodal foundation models for image and video understanding. CoRR, abs/2501.131… — [arXiv:2501.13106](https://arxiv.org/abs/2501.13106) find
  48. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. 2024a. MATHVERSE: does your multi-modal LLM truly see the diagrams in visual math problems? In ECCV (8), Lecture Notes in Computer Science, pages 169–…
  49. Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Shanghang Zhang, Peng Gao, and Hongsheng Li. 2025b. MAVIS: mathematical visual instruction tuning with an automatic data engine. In ICLR. Open-Review.net. find
  50. Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, and Yueting Zhuang. 2024b. Multimodal self-instruct: Synthetic abstract image and visual reasoning instruction using language model. In EMNLP, pages 19228–19252. Association… find
  51. Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. 2025c. Multiple heads are better than one: Mixture of modality knowledge experts for entity representation learning. In ICLR. OpenReview.net. find
  52. Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Min Zhang, Wen Zhang, and Huajun Chen. 2025d. Abstractive visual understanding of multi-modal structured knowledge: A new perspective for mllm evaluation. Preprint, arXiv:2506.01293. — [arXiv:2506.01293](https://arxiv.org/abs/2506.01293) find
  53. Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Wen Zhang, and Huajun Chen. 2024c. Making large language models perform better in knowledge graph completion. In ACM Multimedia, pages 233–242. ACM. find
  54. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren’s song in the AI ocean: A survey on hallucination in large language models. CoRR, abs/2309.01219.
  55. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
  56. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Preprint, arXiv:2304.10592.
  57. Hongyan Zhu, Shuai Qin, Min Su, Chengzhi Lin, Anjie Li, and Junfeng Gao. 2024. Harnessing large vision and language models in agriculture: A review. CoRR, abs/2407.19679. — [arXiv:2407.19679](https://arxiv.org/abs/2407.19679) find
  58. Xun Zhu, Zheng Zhang, Xi Chen, Yiming Shi, Miao Li, and Ji Wu. 2025. Connector-s: A survey of connectors in multi-modal large language models. CoRR, abs/2502.11453. — [arXiv:2502.11453](https://arxiv.org/abs/2502.11453) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-structured-abstractive-reasoning-multi-modal/