Elamur Research⌕
☰
home / papers / 2026 / SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity

SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity

Paper · 2026 · Ishani Mondal, Meera Bharadwaj, Ayush Roy et al.

Summary

Current multimodal models handle local edits to natural photos well. They struggle with structured visuals such as scientific posters and webpages, where one high-level request (for example, adding a section) forces changes across the whole layout. The authors call an edit "SMART" when it does four things:

  • follows the instruction
  • keeps a logical section order
  • keeps related elements consistent with each other
  • maintains clean spatial design with no overlaps, little wasted space and grid alignment

They release SMARTEdit-Bench, a benchmark of implicit, multi-step edits. It is built from GitHub webpage commits, Design2Code and SciPostLayout.

They also propose SMART-Editor, a multi-agent system:

  • An Actor converts the instruction into symbolic layout operations, which an Executor applies.
  • A Critique agent scores the result on reward dimensions and writes natural-language feedback.
  • An Optimizer either refines the plan iteratively at inference time (Reward-Refine) or fine-tunes the model on reward-filtered preference pairs (RewardDPO).

Automatic and human evaluations show more coherent, better-aligned edits than single-shot LLMs and VLMs such as GPT-4o. This matters for design tools that must preserve communicative structure.

Key points

  • Baseline models (LLaMA, Qwen, LLaVA, GPT-4o, Gemini-pro) follow instructions on the surface. However, their narrative coherence falls by more than 15% when sections are reordered, and alignment suffers on insertions and deletions.
  • After manually reviewing 200 real-world edits, the authors grouped them into five categories: adding elements (Insertion), removing them (Deletion), changing their order (Reordering), swapping one object for another (Object Replacement) and clustering related items (Content Grouping).
  • The webpage portion of SMARTEdit-Bench contains:
    • 564 real triplets mined from github.io commit snapshots
    • 510 synthetic triplets derived from Design2Code
  • The poster portion contains:
    • 1,200 synthetic edits from SciPostLayout
    • a gold test set of 250 human-edited posters
    • 2,000 additional synthetic training edits
  • The authors list three limitations: the system relies on dependable reward functions, it is tested only on single-turn edits, and Reward-Refine adds inference cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksComputer Graphics and Computer-Aided Design · 7 works

Cite

Cite as Ishani Mondal, Meera Bharadwaj, Ayush Roy, Aparna Garimella and Jordan Boyd-Graber. 2026. SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.169

@inproceedings{mondal2026smart,
  title = {{SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity}},
  author = {Ishani Mondal, Meera Bharadwaj, Ayush Roy, Aparna Garimella and Jordan Boyd-Graber},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.169},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.169}
}

References

40 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinbin Bai, Wei Chow, Ling Yang, Xiangtai Li, Juncheng Li, Hanwang Zhang, and Shuicheng Yan. 2025. Humanedit: A high-quality human-rewarded dataset for instruction-based image editing. Preprint, arXiv:2412.04280. — [arXiv:2412.04280](https://arxiv.org/abs/2412.04280) find
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  3. Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18392–18402. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  5. SiXiang Chen, Jianyu Lai, Jialin Gao, Tian Ye, Haoyu Chen, Hengyu Shi, Shitong Shao, Yunlong Lin, Song Fei, Zhaohu Xing, Yeying Jin, Junfeng Luo, Xiaoming Wei, and Lei Zhu. 2025. PosterCraft: Rethinking high-quality aesthetic poster generation in a unified framework. Preprint, arXiv:2506.10741. — [arXiv:2506.10741](https://arxiv.org/abs/2506.10741) find
  6. Gemini Team et al. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.
  7. OpenAI et al. 2024b. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  8. Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2023. LayoutGPT: compositional visual planning and generation with large language models. In Proceedings of the 37th International Conference on Neural Information Processin… find
  9. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A generative model for code infilling and synthesis. Preprint, arXiv:2204.05999. find
  10. Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. 2024. LLM Blueprint: Enabling text-to-image generation with complex and detailed prompts. Preprint, arXiv:2310.10640. — [arXiv:2310.10640](https://arxiv.org/abs/2310.10640) find
  11. Claire Golomb. 1987. The development of compositional strategies in children’s drawings. Visual Arts Research, pages 42–52. find
  12. HsiaoYuan Hsu and Yuxin Peng. 2025. PosterO: Structuring layout trees to enable language models in generalized content-aware layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8117– 8127. find
  13. Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. 2024. HQ-Edit: A high-quality dataset for instruction-based image editing. Preprint, arXiv:2404.09990. — [arXiv:2404.09990](https://arxiv.org/abs/2404.09990) find
  14. Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. 2023. LayoutDM: Discrete Diffusion Model for Controllable Layout Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10167–10176. find
  15. Sofia Jamil, Kotla Sai Charan, Sriparna Saha, Koustava Goswami, and Joseph K J. 2025. Do It Yourself (DIY): Modifying images for poems in a zero-shot setting using weighted prompt manipulation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. find
  16. Peidong Jia, Chenxuan Li, Yuhui Yuan, Zeyu Liu, Yichao Shen, Bohan Chen, Xingru Chen, Yinglin Zheng, Dong Chen, Ji Li, Xiaodong Xie, Shanghang Zhang, and Baining Guo. 2024. COLE: A hierarchical generation framework for multi-layered and editable graphic design. Preprint, arXiv:2311.16974. — [arXiv:2311.16974](https://arxiv.org/abs/2311.16974) find
  17. Yeonjoon Jung, Seungtaek Choi, and Seung-won Hwang. 2025. Overcoming source object grounding for semantic image editing. Transactions of the Association for Computational Linguistics, 13:1171– 1185. find
  18. Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. 2019. LayoutVAE: Stochastic scene layout generation from a label set. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). find
  19. Dayeon Ki, Tianyi Zhou, Marine Carpuat, Gang Wu, Puneet Mathur, and Viswanathan Swaminathan. 2025. GraphicBench: A planning benchmark for graphic design with language agents. Preprint, arXiv:2504.11571. — [arXiv:2504.11571](https://arxiv.org/abs/2504.11571) find
  20. Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating graphic layouts with wireframe discriminators. Preprint, arXiv:1901.06767. — [arXiv:1901.06767](https://arxiv.org/abs/1901.06767) find
  21. Jingyu Li, Wei Liu, and Tan Lee. 2022. Edit-net: A lightweight network for unsupervised domain adaptation in speaker verification. Preprint, arXiv:2206.07548. — [arXiv:2206.07548](https://arxiv.org/abs/2206.07548) find
  22. Jiawei Lin, Jiaqi Guo, Shizhao Sun, Zijiang Yang, Jian-Guang Lou, and Dongmei Zhang. 2023. Layout-Prompter: Awaken the design ability of large language models. In NeurIPS 2023. find
  23. Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang, and Ming-Hsuan Yang. 2024. Text-driven image editing via learnable regions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7059–7068. find
  24. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Preprint, arXiv:2304.08485.
  25. George A Lozano. 2014. Ethics of using language editing services in an era of digital communication and heavily multi-authored papers. Science and Engineering Ethics, 20(2):363–377. find
  26. Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. 2023. Drag Your GAN: Interactive point-based manipulation on the generative image manifold. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings,… find
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.
  28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct Preference Optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
  29. Hengyu Shi, Junhao Su, Huansheng Ning, Xiaoming Wei, and Jialin Gao. 2025. LayoutCoT: Unleashing the deep reasoning potential of large language models for layout generation. Preprint, arXiv:2504.10829. — [arXiv:2504.10829](https://arxiv.org/abs/2504.10829) find
  30. Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent Y. F. Tan, and Song Bai. 2024. DragDiffusion: Harnessing diffusion models for interactive point-based image editing. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8839–8849. find
  31. Wanrong Zhu, Ruiyi Zhang, Jennifer Healey, William Yang Wang, and Tong Sun. 2024. Automatic layout planning for visually-rich documents with instruction-following models. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pages 167–172. find
  32. Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2025. Design2Code: Benchmarking multimodal code generation for automated front-end engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics:…
  33. Manan Suri, Puneet Mathur, Franck Dernoncourt, Rajiv Jain, Vlad I Morariu, Ramit Sawhney, Preslav Nakov, and Dinesh Manocha. 2024. DocEdit-v2: Document structure editing via multimodal LLM grounding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 154… find
  34. Shohei Tanaka, Hao Wang, and Yoshitaka Ushiku. 2024. SciPostLayout: A dataset for layout analysis and layout generation of scientific posters. Preprint, arXiv:2407.19787. — [arXiv:2407.19787](https://arxiv.org/abs/2407.19787) find
  35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  36. Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. 2023. InstructEdit: Improving automatic masks for diffusion-based image editing with user instructions. Preprint, arXiv:2305.18047. — [arXiv:2305.18047](https://arxiv.org/abs/2305.18047) find
  37. Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. 2024. GenArtist: Multimodal llm as an agent for unified image generation and editing. Preprint, arXiv:2407.05600. — [arXiv:2407.05600](https://arxiv.org/abs/2407.05600) find
  38. Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, and Ran Xu. 2024. HIVE: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec… find
  39. Zhao Zhang, Yutao Cheng, Dexiang Hong, Maoke Yang, Gonglei Shi, Lei Ma, Hui Zhang, Jie Shao, and Xinglong Wu. 2025a. CreatiPoster: Towards editable and controllable multi-layer graphic design generation. Preprint, arXiv:2506.10890. — [arXiv:2506.10890](https://arxiv.org/abs/2506.10890) find
  40. Zhilin Zhang, Xiang Zhang, Jiaqi Wei, Yiwei Xu, and Chenyu You. 2025b. PosterGen: Aesthetic-aware paper-to-poster generation via multi-agent llms. Preprint, arXiv:2508.17188. — [arXiv:2508.17188](https://arxiv.org/abs/2508.17188) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mondal-2026-smart-editor-multi-agent-framework/