Elamur Research⌕
☰
home / papers / 2026 / SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing

SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing

Paper · 2026 · Tong Zhang, Honglin Lin, Zhou Liu et al.

Summary

Text-to-image systems can produce scientific figures that look polished but are logically wrong: they skip steps, invent labels, or point arrows in the wrong direction. Current benchmarks miss these errors because they score image similarity or subjective quality, or because they check intermediate code rather than the rendered picture. SciFlow-Bench addresses this by judging generated diagrams on whether their structure can be recovered from the pixels. The authors collected framework figures from real scientific PDFs and paired each one with a canonical ground-truth graph of components and directed links. During evaluation, models are treated as black boxes. Each output image is parsed back into a graph and compared with the reference in a round-trip setup. Both the reference graphs and the inverse parsing come from the same hierarchical multi-agent system, which has planning, perception, and reasoning layers. The experiments cover code-driven, diffusion, and autoregressive multimodal generators. Across all of them, visual quality and structural correctness diverge, and the gap is largest for diagrams with complex topology. This makes structure-aware evaluation necessary for scientific multimodal generation.

Key points

  • Benchmark contents: 500 real-world framework diagrams from five research domains, each paired with a ground-truth graph built by a hierarchical multi-agent system (HMAS).
  • Agents in the HMAS:
    • Methodologist and Visual Translator handle planning.
    • Shape Hunter, Text Spotter, and Fusion Arbiter handle perception.
    • Topology Coder and Graph Architect handle structural reasoning.
    • The agents share a JSON blackboard.
  • Scoring: The benchmark combines three levels into an overall score:
    • graph-level node/edge F1;
    • text-level coverage, faithfulness, and alignment;
    • image-level measures such as CLIP and LPIPS.
  • Planning-layer ablation (structured vs. naive prompts):
    • Structured prompts beat naive ones for every generator tested.
    • Gemini 3 Pro Image reached the highest overall score (0.311), ahead of Graphviz (GPT-4o→DOT) at 0.283.
    • SDXL and PixArt-Σ stayed below 0.1 overall, with graph scores around 0.01–0.02.
  • Parser ablation:
    • Dropping the Text Spotter cut node recall from 0.96 to 0.48.
    • Removing the Shape Hunter caused under-segmentation and a sharp loss in edge recall.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🏢Huawei TechnologiesShenzhen · China96 works · 15,432 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Tong Zhang, Honglin Lin, Zhou Liu, Chong Chen and Wentao Zhang. 2026. SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.807

@inproceedings{zhang2026sciflow,
  title = {{SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing}},
  author = {Tong Zhang, Honglin Lin, Zhou Liu, Chong Chen and Wentao Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.807},
  url = {https://doi.org/10.18653/v1/2026.acl-long.807}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, and 1 others. 2023. Improving image generation with better captions. Computer Science. https://cdn.openai. com/papers/dall-e-3. pdf, 2(3):8. ↗
  2. Yifan Chang, Yukang Feng, Jianwen Sun, Jiaxin Ai, Chuanhao Li, S Kevin Zhou, and Kaipeng Zhang. 2025. Sridbench: Benchmark of scientific research illustration drawing of image generation model. arXiv preprint arXiv:2505.22126. — [arXiv:2505.22126](https://arxiv.org/abs/2505.22126) find
  3. Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. 2023. Textdiffuser: Diffusion models as text painters. Advances in Neural Information Processing Systems, 36:9353–9387. find
  4. Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. 2024. Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74–91. Springer. find
  5. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. find
  6. Yi-Chun Chen. 2025. Structured graph representations for visual narrative reasoning: A hierarchical framework for comics. arXiv preprint arXiv:2506.10008. — [arXiv:2506.10008](https://arxiv.org/abs/2506.10008) find
  7. Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. Visual programming for step-by-step text-to-image generation and evaluation. Advances in Neural Information Processing Systems, 36:6048–6069. find
  8. Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. 2024. Ranni: Taming text-to-image diffusion for accurate instruction following. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4744– 4753. find
  9. Xinbo Gao, Bing Xiao, Dacheng Tao, and Xuelong Li. 2010. A survey of graph edit distance. Pattern Analysis and applications, 13(1):113–129. find
  10. Ziyi Guo, Zhou Liu, and Wentao Zhang. 2025. Paper2sysarch: Structure-constrained system architecture generation from scientific papers. arXiv preprint arXiv:2511.18036. — [arXiv:2511.18036](https://arxiv.org/abs/2511.18036) find
  11. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7514– 7528.
  12. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30. find
  13. Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747. find
  14. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678. find
  15. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008.
  16. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.
  17. Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. 2023. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655. find
  18. Chumeng Liang and Jiaxuan You. 2025. Diagrameval: Evaluating llm-generated diagrams via graphs. arXiv preprint arXiv:2510.25761. — [arXiv:2510.25761](https://arxiv.org/abs/2510.25761) find
  19. Honglin Lin, Chonghan Qin, Zheng Liu, Qizhi Pei, Yu Li, Zhanping Zhong, Xin Gao, Yanfeng Wang, Conghui He, and Lijun Wu. 2026. Scientific image synthesis: Benchmarking, methodologies, and downstream utility. arXiv preprint arXiv:2601.17027. — [arXiv:2601.17027](https://arxiv.org/abs/2601.17027) find
  20. Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125.
  21. Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer. find
  22. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pages 2263– 2279.
  23. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  24. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the ieee/cvf winter conference on applications of computer vision, pages 1527–1536. find
  25. Nii H Penny. 1986. Blackboard systems: The blackboard model of problem solving and the evolution of blackboard architectures. The AI Magazine. find
  26. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.
  27. Yingjie Qi, Jianlei Yang, Yiou Wang, Yikun Wang, Dayu Wang, Ling Tang, Cenlin Duan, Xiaolin He, and Weisheng Zhao. 2025. Cimflow: An integrated framework for systematic design and evaluation of digital cim architectures. arXiv preprint arXiv:2505.01107. — [arXiv:2505.01107](https://arxiv.org/abs/2505.01107) find
  28. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
  29. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, and 1 others. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processin…
  30. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. Advances in neural information processing systems, 29.
  31. Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pages 70–80. find
  32. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  33. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091. find
  34. Zihang Wang, Yilun Zhao, Kaiyan Zhang, Chen Zhao, Manasi Patwardhan, and Arman Cohan. 2025. Scisketch: An open-source framework for automated schematic diagram generation in scientific papers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstra… find
  35. Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, and 1 others. 2025. Qwen-image technical report. arXiv preprint arXiv:2508.02324.
  36. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, and 1 others. 2024. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling. find
  37. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  38. Zhang Zhang, Qiang Zhang, Wei Cui, Shuai Shi, Yijie Guo, Gang Han, Wen Zhao, Jingkai Sun, Jiahang Cao, Jiaxu Wang, and 1 others. 2025. Occupancy world model for robots. arXiv preprint arXiv:2505.05512. — [arXiv:2505.05512](https://arxiv.org/abs/2505.05512) find
  39. George Kingsley Zipf. 2016. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio books. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-sciflow-bench-evaluating-structure-aware/