Elamur Research⌕
☰
home / papers / 2026 / TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

Paper · 2026 · Chao Wen, Jacqueline Staub and Adish Singla

Summary

The paper investigates how well vision-language models handle visual programming in educational settings. Prior work has mostly targeted productivity domains such as UIs, SVGs or charts. The authors present TURTLEAI, a Turtle Graphics benchmark of 823 tasks derived from XLogoOnline, a Logo-based platform used by students in grades 3–6. Each task gives the model a target image and asks it to write a Python draw(t) function that reproduces the pattern. This requires recognizing shapes, reasoning about angles, scale and layout, and expressing that structure in code. An evaluation framework, TURTLEAI-Eval, judges correctness independently of size, position and line width. More than 20 VLMs, including GPT-5, GPT-4o and Qwen2-VL-72B, were tested, and most solved fewer than 30% of tasks. The authors also introduce TURTLEAI-Datagen, which expands a handful of seed tasks into synthetic image–code examples. Further training of Qwen2-VL-72B on these generated examples improved its real-world results by about 20 points. Error analysis indicates that the gain comes mainly from better consistency between visual reasoning and the generated code, not from better perception. The benchmark therefore exposes a gap that matters for using VLMs in classrooms.

Key points

  • Benchmark structure: TURTLEAI-DS contains three datasets:
    • DSReal: 102 expert-designed XLogoOnline tasks with manually written Python solutions.
    • DSCraft: hand-drawn inputs.
    • DSSyn: synthetic tasks.
  • Task variety: tasks span six categories (basic geometry, translation, rotation, scaling, spiral, composite) and three difficulty levels.
  • Fine-tuning gains: fine-tuning lifted Qwen2-VL-72B (Qwen2-VL-72B-TURTLE) from 10.8% to 35.3% success in the failure analysis. Programming errors dropped from 30.4% to 6.9%, while visual understanding and spatial reasoning errors stayed about the same.
  • Code compactness: code length was measured as a ratio to the reference solution. Base Qwen2-VL-72B produced very verbose code when it failed (ratio 4.01), whereas the fine-tuned model stayed near 1.0. GPT-4o held a steady ratio of 1.25.
  • Comparison with TurtleBench: TurtleBench evaluated 3 VLMs on 260 manually created tasks. Fine-tuned models in this work still have trouble with hand-drawn inputs. Code and data are available at github.com/machine-teaching-group/acl2026-turtleai.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität TrierTrier · Germany9 works · 9 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Chao Wen, Jacqueline Staub and Adish Singla. 2026. TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.334

@inproceedings{wen2026turtleai,
  title = {{TURTLEAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics}},
  author = {Chao Wen, Jacqueline Staub and Adish Singla},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.334},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.334}
}

References

58 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas,… — Pixtral 12B
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-VL Technical Report. CoRR, abs/2…
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-VL Technical Report. CoRR, abs/2502.1392…
  4. Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. 2024. DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ. In NeurIPS. find
  5. Kevin Ellis, Catherine Wong, Maxwell I. Nye, Mathias Sablé-Meyer, Lucas Morales, Luke B. Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2021. DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning. In PLDI. find
  6. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In KDD. find
  7. Shuchi Grover and Roy Pea. 2013. Computational thinking in K–12: A review of the state of the field. Educational researcher. find
  8. Yi Gui, Yao Wan, Zhen Li, Zhongyi Zhang, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, and Xiangliang Zhang. 2025. UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs. In WWW. find
  9. Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual Programming: Compositional visual reasoning without training. In CVPR.
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR.
  11. Blockly. 2022. Games for Tomorrow’s Programmers. https://blockly.games/. ↗
  12. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR.
  13. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. CoRR, abs/2312.14238.
  14. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In SIGOPS.
  15. CodeHS. 2025. Turtle Graphics with Tracy the Turtle. https://codehs.com/hourofcode/tracy. Accessed: 2025-09-01. ↗
  16. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. LLaVA-OneVision: Easy Visual Task Transfer. CoRR, abs/2408.03326.
  17. Code.org. 2013. Code.org: Learn Computer Science. https://code.org/. ↗
  18. Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. NVLM: Open Frontier-Class Multimodal LLMs. CoRR, abs/2409.11402.
  19. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, and 32 others. 2024.… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  20. Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. 2024b. Revisiting Catastrophic Forgetting in Large Language Model Tuning. In EMNLP (Findings). find
  21. Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, and Jing Ma. 2024c. MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems. In EMNLP (Findings). find
  22. Wen-Ding Li and Kevin Ellis. 2024. Is Programming by Example Solved by LLMs? In NeurIPS. find
  23. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by Chat-GPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In NeurIPS. find
  24. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In ICLR.
  25. John H. Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch Programming Language and Environment. ACM Trans. Comput. Educ. find
  26. Jeremy Marbach, Alexandra Maximova, and Jacqueline Staub. 2022. A Tool to Create and Conduct Custom Assessments in Turtle Graphics. In ISSEP. 6 others. 2023. Code Llama: Open Foundation Models for Code. CoRR, abs/2308.12950.
  27. Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2025. Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering. In NAACL.
  28. Adish Singla. 2023. Evaluating ChatGPT and GPT-4 for Visual Programming. In ICER - Volume 2. find
  29. Jacqueline Staub. 2021. Logo Environments in the Focus of Time. Bulletin of EATCS. https://openai.com/ ↗
  30. Turtle Academy. 2025. Turtle Academy. https://turtleacademy.com/. Accessed: 2025-09-01. ↗
  31. OpenAI. 2024b. GPT-4v. https://openai.com/ index/gpt-4v-system-card/. ↗
  32. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the W…
  33. OpenAI. 2024a. GPT-4o. index/hello-gpt-4o/. find
  34. GPT-5. https://openai.com/ ↗
  35. OpenAI. 2025b. o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. ↗
  36. Victor-Alexandru Padurean and Adish Singla. 2024. Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming. In NeurIPS Track on Datasets and Benchmarks. find
  37. Roy D Pea. 1987. Logo programming and problem solving. find
  38. Python. 2024. Python Turtle Graphics. https://docs.python.org/3/library/turtle.html. ↗
  39. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML.
  40. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In ICML.
  41. Sina Rismanchian, Yasaman Razeghi, Sameer Singh, and Shayan Doroudi. 2025. TurtleBench: A Visual Programming Benchmark in Turtle Geometry. In NAACL. find
  42. Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Sai Rajeswar, David Vázquez, Christopher Pal, and Marco Pedersoli. 2025. StarVector: Generating Scalable Vector Graphics Code from Images and Text. In AAAI. find
  43. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, and Jas… — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  44. Chao Wen, Ahana Ghosh, Jacqueline Staub, and Adish Singla. 2024. Task Synthesis for Elementary Visual Programming in XLogoOnline Environment. In AIED Track on Late Breaking Results. find
  45. Chao Wen, Tung Phung, Pronita Mehrotra, Sumit Gulwani, Roger E. Beaty, Tomohiro Nagashima, and Adish Singla. 2025a. Exploration vs. Fixation: Scaffolding Divergent and Convergent Thinking for Human-AI Co-Creation with Generative Models. CoRR, abs/2512.18388. — [arXiv:2512.18388](https://arxiv.org/abs/2512.18388) find
  46. Chao Wen, Jacqueline Staub, and Adish Singla. 2025b. Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment. In ACL. find
  47. Chengyue Wu, Zhixuan Liang, Yixiao Ge, Qiushan Guo, Zeyu Lu, Jiahao Wang, Ying Shan, and Ping Luo. 2025. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. In NAACL (Findings). find
  48. XLogoOnline. 2024. XLogoOnline Platform. https://xlogo.inf.ethz.ch/. ↗
  49. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions. In ICLR.
  50. Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, and Yujiu Yang. 2025. ChartMimic: Evaluating LMM’s Cross-Modal Reasoning Capability via Chart-to-Code Generation. In ICLR. find
  51. Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. MMMU: A Massive Multi-Discipline Multimodal U…
  52. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. STaR: Bootstrapping Reasoning With Reasoning. In NeurIPS.
  53. Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2024a. AgentTuning: Enabling Generalized Agent Abilities for LLMs. In ACL (Findings).
  54. Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, and 36 others. 2024b. ChatGLM: A Family of Large Language Models from GLM-130B…
  55. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. In ACL.
  56. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, and 32 others. 2025. InternVL3: Exploring Advanced Training and Test-…
  57. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 2 others. 2025. Big-CodeBench… find
  58. Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. 2024. VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation. In EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wen-2026-turtleai-benchmarking-multimodal-models-visual/