Elamur Research⌕
☰
home / papers / 2026 / Frame2KG: A Benchmark and Evaluation Toolkit for Interpretable Frame-to-Graph Generation

Frame2KG: A Benchmark and Evaluation Toolkit for Interpretable Frame-to-Graph Generation

Summary

The paper targets a missing resource: there is no open, reproducible benchmark for turning a single image into a localised knowledge graph that is checked both for correctness and for efficiency. Such graphs are meant for robots and other devices that run on-device. The authors release Frame2KG-YC2, a synthetic corpus built from YouCook2 cooking videos. Each sampled keyframe is paired with a strict-JSON graph and a short graph_sentence paraphrase. In the graph, nodes are typed objects with normalised boxes and confidences, and edges are predicates. Labels come from a GPT vision–language teacher, and outputs failing schema checks are regenerated. As baselines, Qwen2.5-VL 3B and 7B are fine-tuned with LoRA in four adapter configurations. A deterministic scoring toolkit first filters node pairs by box overlap, then assigns them one-to-one with Hungarian matching on combined spatial and text similarity. The best runs reach Node F1 µ of 0.621 and Edge F1 µ of 0.208, with over 98% schema conformity. These results show that compact, quantised VLMs can produce machine-readable, interpretable scene graphs.

Key points

  • Data pipeline: frames are taken from YouCook2 (2,000 videos, 89 recipes) at 1 fps. Near-duplicates are removed with OpenCLIP ViT-B/32 cosine similarity (τ = 0.90). Batched JSONL annotation cut API cost by about half.
  • Adapters: LoRA is applied to attention projections (QKVO), with or without GateProj/Up/Down MLP projections. This trains under 0.5% of model weights.
  • Effect of gating (3B): Edge F1 µ rose from 0.187 to 0.208, and mean matched IoU from 0.600 to 0.609. The larger 7B backbone mainly helps entity naming.
  • Recommended deployment choice: 3B/QKVO-Gate scores Node F1 µ 0.616, close to 0.621 for 7B. It is equal or better on edges and IoU and needs much less memory.
  • Quantisation and predicate matching: INT4 post-training quantisation costs little (val_dev Node F1 0.664 → 0.658). Embedding-based predicate matching roughly doubled compute without improving accuracy, so normalised string equality is the default.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ys6kofrzoc5

@inproceedings{watson2026frame2kg,
  title = {{Frame2KG: A Benchmark and Evaluation Toolkit for Interpretable Frame-to-Graph Generation}},
  author = {Watson, Lewis N. and Strathearn, Carl and Mitchell, Kenny and Yu, Yanchao},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4ys6kofrzoc5},
  url = {https://doi.org/10.63317/4ys6kofrzoc5}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil… find our card
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Ha… find our card
  3. Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, Tanvi Dinkar, Ondřej Dušek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier González-Corbelle, Dirk Hovy, M… find
  4. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel… find
  5. Anthony Brohan et al. 2023b. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning (CoRL), volume 229 of Proceedings of Machine Learning Research (PMLR), pages 2165–2183. PMLR. find
  6. Guikun Chen, Jin Li, and Wenguan Wang. 2024. Scene graph generation with role-playing large language models. find
  7. Open X-Embodiment Collaboration. 2023. Open x-embodiment: Robotic learning datasets and rt-x models. CoRR, abs/2310.08864. find
  8. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS). find our card
  9. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. find
  10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient fine-tuning of quantized llms. In Advances in Neural Information Processing Systems (NeurIPS). find
  11. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zen… find
  12. Zeeshan Hayder and Xuming He. 2024. Dsgg: Dense relation transformer for an end-to-end scene graph generation. find
  13. Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. 2022. Towards open-vocabulary scene graph generation with prompt-based finetuning. find
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). find
  15. Siyuan Huang, Zhi-Qi Cheng, Karl Schmeckpeper, Shuran Zhang, Anima Anandkumar, Yuke Zhu, and Zoran Popović. 2023. Conceptgraphs: Open-vocabulary 3d scene graphs for language-grounded manipulation. In Proceedings of the 7th Conference on Robot Learning (CoRL), volume 229 of Proceedings of Machine Lea… find
  16. Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. 2024. Egtr: Extracting graph from transformer for scene graph generation. find
  17. Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2758–2766. find
  18. Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. 2020. Action genome: Actions as compositions of spatio-temporal scene graphs. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10233–10244. find
  19. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journ… find our card
  20. Harold W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83–97. find
  21. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. 2018. Tvqa: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1369–1379. find our card
  22. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR. find
  23. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML). find our card
  24. Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. 2024. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. find
  25. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft coco: Common objects in context. find our card
  26. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS). find our card
  27. Julian Lorenz, Robin Schön, Katja Ludwig, and Rainer Lienhart. 2024. A review and efficient implementation of scene graph generation metrics. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2567–2575. find
  28. OpenAI. 2025. Openai o3 and o4-mini system card. Technical report, OpenAI. Accessed: Oct. 16, 2025. find
  29. Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML. find our card
  30. Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, and Luca Carlone. 2020. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. In Robotics: Science and Systems (RSS). find
  31. Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. 2017. Video visual relation detection. In Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, page 1300–1308, New York, NY, USA. Association for Computing Machinery. find
  32. Luowei Zhou, Nathan Louis, and Jason J. Corso. 2018. Weakly-supervised video object grounding from text by loss weighting and object interaction. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/watson-2026-frame2kg-benchmark-evaluation-toolkit-interpretable/