Elamur Research⌕
☰
home / papers / 2026 / Expert-Guided Schema-Based Structured Extraction from CONSORT Diagrams Using Vision-Language Models

Expert-Guided Schema-Based Structured Extraction from CONSORT Diagrams Using Vision-Language Models

Paper · 2026 · Damian Stachura, Bartosz Przechera, Monika Opałek et al.

Summary

The paper addresses turning CONSORT flow diagrams into structured, machine-readable data. Such diagrams chart each stage a participant goes through in a randomized controlled trial, from initial screening and random allocation to follow-up and final analysis. Pulling data from them requires more than OCR: a model must also recover layout, grouping and how boxes connect. The authors, from Evidence Prime, built a 200-diagram benchmark from CC-BY PubMed Central articles. Three biomedical experts annotated it against a nested schema derived from the CONSORT literature, with one study-level object and per-arm objects. They then tested proprietary and open-weight VLMs in a zero-shot setting, comparing single-pass schema filling under different prompts with stepwise decomposition. Scoring used metrics that account for the nested structure. Single-pass extraction with an expert-written prompt gave the best results for frontier proprietary models, and Gemini 3 Pro was the strongest overall. Stepwise prompting mainly helped smaller open-weight models on arm-level details. Even the best configuration fully reproduced only about 70% of diagrams. The task is therefore practical but not solved, and the results give concrete guidance on which strategy suits which class of model in evidence-synthesis pipelines.

Key points

  • Benchmark: 180 diagrams were curated for visual and structural diversity from a random pool of 500 PubMed Central RCT figures. Another 20 expert-chosen, hard-to-agree-on cases were added as a stress-test subset. Disagreements during annotation were resolved by an expert panel.

  • Evaluation suite: the metrics cover group-level F1, leaf micro F1, strict exact field accuracy, whole-diagram exact match, and diagnostic error measures.

  • Best configuration (expert-guided single pass):

    • Group F1: 0.976
    • Leaf micro F1: 0.971
    • Exact field accuracy: 0.944
    • Exact match: 0.705

  • Stepwise prompting for open-weight models:

    • Qwen3-VL 30B: leaf micro F1 rose from 0.495 to 0.611.
    • Gemma-4-26B-A4B-it: exact field accuracy rose from 0.068 to 0.407.

  • Common errors:

    • Ambiguous or merely implied counts in the source diagrams.
    • Nested randomization and partial assessments.
    • Qwen3-VL 8B filling unreported fields with zeros.
    • Verbose rather than normalized arm names.
    • Subscript and superscript misreads.

    The authors propose arm-localized, crop-based extraction as a next step.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksDecision Sciences · 249 worksStatistics, Probability and Uncertainty · 65 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Damian Stachura, Bartosz Przechera, Monika Opałek, Ewelina Sadowska, Ewa Borowiack and Artur Nowak. 2026. Expert-Guided Schema-Based Structured Extraction from CONSORT Diagrams Using Vision-Language Models. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.77

@inproceedings{stachura2026expert,
  title = {{Expert-Guided Schema-Based Structured Extraction from CONSORT Diagrams Using Vision-Language Models}},
  author = {Damian Stachura, Bartosz Przechera, Monika Opałek, Ewelina Sadowska, Ewa Borowiack and Artur Nowak},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.77},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.77}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul Arbaz, Heng Fan, Junhua Ding, Meikang Qiu, and Yunhe Feng. 2024. Genflowchart: Parsing and understanding flowchart using generative ai. In Knowledge Science, Engineering and Management: 17th International Conference, KSEM 2024, Birmingham, UK, August 16–18, 2024, Proceedings, Part I, volume 14… find
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  3. Yufan Chen, Ching Ting Leung, Bowen Yu, Jianwei Sun, Yong Huang, Linyan Li, Hao Chen, and Hanyu Gao. 2025. A multi-agent system enables versatile information extraction from the chemical literature. arXiv preprint arXiv:2507.20230. — [arXiv:2507.20230](https://arxiv.org/abs/2507.20230) find
  4. Google DeepMind. 2025. Gemini 3: Flash and pro models. https://deepmind.google. ↗
  5. Pritam Deka and Barry Devereux. 2025. Structured extraction from business process diagrams using vision-language models. arXiv preprint arXiv:2511.22448. — [arXiv:2511.22448](https://arxiv.org/abs/2511.22448) find
  6. Masato Fujitake. 2024. Layoutllm: Large language model instruction tuning for visually rich document understanding. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10219–10224, Torino, Italia. ELRA an… find
  7. Madhusudan Ghosh, Shrimon Mukherjee, Asmit Ganguly, Partha Basuchowdhuri, Sudip Kumar Naskar, and Debasis Ganguly. 2024. Alpapico: Extraction of pico frames from clinical trial documents using llms. Methods, 226:78–88. find
  8. Google DeepMind. 2026. Gemma 4 model card. https://ai.google.dev/gemma/docs/ core/model_card_4. Accessed: 2026-05-20. ↗
  9. Linh Hoang, Yingjun Guan, and Halil Kilicoglu. 2023. Methodological information extraction from randomized controlled trial publications: a pilot study. In AMIA Annual Symposium Proceedings, pages 542– 551. find
  10. Sally Hopewell, An-Wen Chan, Gary S Collins, Hróbjartsson A., D. Moher, K. F. Schulz, and 1 others. 2025. find
  11. Consort 2025 statement: updated guideline for reporting randomised trials. BMJ, 388:e081123. find
  12. Sally Hopewell, Allison Hirst, Gary S. Collins, Sue Mallett, Ly-Mee Yu, and Douglas G. Altman. 2011. Reporting of participant flow diagrams in published reports of randomized trials. Trials, 12:253. find
  13. Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. 2023. mplug-paperowl: Scientific diagram analysis with the multimodal large language model. arXiv preprint arXiv:2311.18248. — [arXiv:2311.18248](https://arxiv.org/abs/2311.18248) find
  14. Changkai Ji, Yang Li, Yingwen Wang, Wen He, Ying Cheng, Yuejie Zhang, Rui Feng, and Xiaobo Zhang. 2026. Evimmqa: Multimodal question answering for medical evidence extraction in systematic reviews. Pattern Recognition, page 113441. Available online 5 March 2026. find
  15. Daehyun Kim and Hong Yu. 2011. extraction in biomedical literature. 6(1):e15338. find
  16. Figure text PLoS ONE, Mathieu Laï-king and Patrick Paroubek. 2025. Evaluation of clinical trials reporting quality using large language models. arXiv preprint arXiv:2510.04338. — [arXiv:2510.04338](https://arxiv.org/abs/2510.04338) find
  17. David Moher, Sally Hopewell, Kenneth F Schulz, Victor Montori, Peter C Gøtzsche, PJ Devereaux, Diana Elbourne, Matthias Egger, and Douglas G Altman. 2010. find
  18. Consort 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ, 340:c869. find
  19. Teresa O’Leary, June Weiss, Benjamin Toll, Cynthia Brandt, and Steven L. Bernstein. 2019. Automated generation of consort diagrams using relational database software. Applied Clinical Informatics, 10(1):60–65. find
  20. Takamitsu Omasa, Ryo Koshihara, and Masumi Morishige. 2025. Arrow-guided vlm: Enhancing flowchart understanding via arrow direction encoding. arXiv preprint arXiv:2505.07864. — [arXiv:2505.07864](https://arxiv.org/abs/2505.07864) find
  21. OpenAI. 2025. Gpt-5 technical report. openai.com. https:// find
  22. Huitong Pan, Qi Zhang, Cornelia Caragea, Eduard Dragut, and Longin Jan Latecki. 2024. Flowlearn: Evaluating large vision-language models on flowchart understanding. In Proceedings of the 27th European Conference on Artificial Intelligence (ECAI 2024), pages 73–80. IOS Press. find
  23. Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2024. Spiqa: A dataset for multimodal question answering on scientific papers. In Advances in Neural Information Processing Systems, volume 37, pages 118807–118833. find
  24. Pouria Rouzrokh, Bardia Khosravi, Parsa Rouzrokh, and Moein Shariatnia. 2025. Lattereview: A multi-agent framework for systematic review automation using large language models. arXiv preprint arXiv:2501.05468. — [arXiv:2501.05468](https://arxiv.org/abs/2501.05468) find
  25. Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, and Dan Roth. 2024. Flowvqa: Mapping multimodal logic in visual question answering with flowcharts. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1330–1350, Bangkok, Thailand.… find
  26. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128.
  27. Simon Tannert, Marcelo G. Feighelstein, Jasmina Bogojeska, Joseph Shtok, Assaf Arbelle, Peter W. J. Staar, Anika Schumann, Jonas Kuhn, and Leonid Karlinsky. 2023. Flowchartqa: The first large-scale benchmark for reasoning over flowcharts. In Proceedings of the 1st Workshop on Linguistic Insights fro… find
  28. Fan Wang, Richard L. Schilsky, David Page, Robert M. Califf, Kei Cheung, Xiaofei Wang, and Herbert Pang. 2020. Development and validation of a natural language processing tool to generate the consort reporting checklist for randomized clinical trials. JAMA Network Open, 3(10):e2014661. find
  29. Fei Wang, Yuewen Zheng, Qin Li, Jingyi Wu, Pengfei Li, and Luxia Zhang. 2024a. Chatschema: A pipeline of extracting structured information with large multimodal models based on schema. arXiv preprint arXiv:2407.18716. — [arXiv:2407.18716](https://arxiv.org/abs/2407.18716) find
  30. Xingbo Wang, Samantha L. Huey, Rui Sheng, Saurabh Mehta, and Fei Wang. 2025. Scidasynth: Interactive structured data extraction from scientific literature with large language model. Campbell Systematic Reviews, 21(4):e70073. find
  31. Zifeng Wang, Lang Cao, Benjamin Danek, Qiao Jin, Zhiyong Lu, and Jimeng Sun. 2024b. Accelerating clinical evidence synthesis with large language models. arXiv preprint arXiv:2406.17755. — [arXiv:2406.17755](https://arxiv.org/abs/2406.17755) find
  32. Junyi Ye, Ankan Dash, Wenpeng Yin, and Guiling Wang. 2024. Beyond end-to-end vlms: Leveraging intermediate text representations for superior flowchart understanding. arXiv preprint arXiv:2412.16420. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stachura-2026-expert-guided-schema-based-structured/