Elamur Research⌕
☰
home / papers / 2026 / SceneLM: 3D-Aware Language Models for Editable 3D Scene Synthesis

SceneLM: 3D-Aware Language Models for Editable 3D Scene Synthesis

Paper · 2026 · Xingbo Yao, Xiaoyu Chen, Doudou Zhang et al.

Summary

SceneLM addresses the problem of building an editable 3D scene from one RGB image. Geometry-based pipelines give good quality but usually have to be rerun to fix errors. LLM tool-using systems are interactive but work mainly from text and lack metric 3D understanding. SceneLM reframes the task as two steps. First, a vision-language model reads the image, plus camera intrinsics when available, and writes a JSON layout. For each object it gives the category, 3D center and size in millimetres, and a yaw value discretized into 15° bins. Second, a deterministic tool suite turns this layout into an actual scene. Users edit by changing layout fields and re-running only the affected tool calls.

The model combines a SigLIP-2 encoder, a Qwen 2B-scale LLM and an extra geometry head. It is trained with a mix of token cross-entropy and a geometric loss, on five curated indoor, outdoor and tabletop datasets converted into one instruction-tuning format. The authors report better layout estimation than open and proprietary MLLMs, and more complete, physically plausible generated scenes.

Key points

  • Output format: a JSON list of objects, each with class, center_mm, size_mm and yaw_bin. Yaw is treated as a 24-way classification target.
  • Ablation results (1,000 test images, F1 at 3D IoU 0.25 / 0.5):
    • Full SceneLM: 58.5 / 36.1
    • Without GeomHead and dual supervision: 47.6 / 28.4
    • Without yaw discretization: 44.1 / 25.1
    • Plain Qwen3-VL-2B trained only with JSON cross-entropy: 18.2 / 12.4
  • Fine-tuning: full fine-tuning (58.5 / 36.1) beats LoRA (51.3 / 32.7).
  • Scene generation versus SceneGen:
    • SceneLM's collided-pairs rate is 0.37, versus 0.23 for SceneGen.
    • SceneGen's lower rate comes partly from producing fewer objects and dropping small items.
    • Qwen3-VL-30B-A3B-Instruct, run through the same tools, often produced incomplete JSON and boxes with wrong scale.
  • Limitations:
    • When intrinsics are missing, default values are used, which can skew absolute scale.
    • Output quality depends on the external asset generator.
    • In complex scenes the model may still emit invalid or incomplete JSON.
    • Planned future work includes export to Habitat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓Hong Kong University of Science and TechnologyHong Kong · China170 works · 26,341 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China67 works · 1,765 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksEngineering · 188 worksComputational Mechanics · 5 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Xingbo Yao, Xiaoyu Chen, Doudou Zhang, Mingzhi Sheng, Boyuan Cao, Ying-Cong Chen and Hui Xiong. 2026. SceneLM: 3D-Aware Language Models for Editable 3D Scene Synthesis. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2116

@inproceedings{yao2026scenelm,
  title = {{SceneLM: 3D-Aware Language Models for Editable 3D Scene Synthesis}},
  author = {Xingbo Yao, Xiaoyu Chen, Doudou Zhang, Mingzhi Sheng, Boyuan Cao, Ying-Cong Chen and Hui Xiong},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2116},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2116}
}

References

38 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. 2021. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7822–7831. find
  2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  3. Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and 1 others. 2021. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897. — [arXiv:2111.08897](https://arxiv.org/abs/2111.08897) find
  4. Aleksey Bokhovkin, Quan Meng, Shubham Tulsiani, and Angela Dai. 2025. Scenefactor: Factored latent 3d diffusion for controllable 3d scene generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 628–639. find
  5. Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. 2023. Omni3d: A large benchmark and model for 3d object detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13154–13164. find
  6. Martin J.J. Bucher and Iro Armeni. 2025. Respace: Text-driven 3d indoor scene synthesis and editing with preference alignment. arXiv preprint arXiv:2506.02459. — [arXiv:2506.02459](https://arxiv.org/abs/2506.02459) find
  7. Jang Hyun Cho, B. Ivanovic, Yulong Cao, Edward Schmerling, Yue Wang, Xinshuo Weng, Boyi Li, Yurong You, Philipp Krahenbuhl, Yan Wang, and Marco Pavone. 2024. Language-image models with 3d understanding. arXiv preprint arXiv:2405.03685. — [arXiv:2405.03685](https://arxiv.org/abs/2405.03685) find
  8. Wei Deng, Mengshi Qi, and Huadong Ma. 2025. Global-local tree search in vlms for 3d indoor scene generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8975–8984. find
  9. Chuan Fang, Heng Li, Yixun Liang, Jia Zheng, Yongsen Mao, Yuan Liu, Rui Tang, Zihan Zhou, and Ping Tan. 2025. Spatialgen: Layout-guided 3d indoor scene generation. arXiv preprint arXiv:2509.14981. — [arXiv:2509.14981](https://arxiv.org/abs/2509.14981) find
  10. Weitao Feng, Hang Zhou, Jing Liao, Li Cheng, and Wenbo Zhou. 2025. Casagpt: cuboid arrangement and scene assembly for interior design. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29173–29182. find
  11. Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(11):1231–1237. find
  12. Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu, Feng Zheng, Lizhuang Ma, and Jiangmiao Pang. 2025. Mesatask: Towards task-driven tabletop scene generation via 3d spatial reasoning. arXiv preprint arXiv:2509.22281. — [arXiv:2509.22281](https://arxiv.org/abs/2509.22281) find
  13. Xiaolu Hou, Mingcheng Li, Dingkang Yang, Jiawei Chen, Ziyun Qian, Xiao Zhao, Yue Jiang, Jinjie Wei, Qingyao Xu, and Lihua Zhang. 2025. Bloomscene: Lightweight structured 3d gaussian splatting for crossmodal scene generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39… find
  14. Yukun Huang, Yanning Zhou, Jianan Wang, Kaiyi Huang, and Xihui Liu. 2025. Dreamcube: 3d panorama generation via multi-plane synchronization. find
  15. Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Philipp Henzler, Konrad Schindler, and Federico Tombari. 2025. Cubediff: Repurposing diffusion-based image models for panorama generation. find
  16. Yuheng Liu, Xinke Li, Yuning Zhang, Lu Qi, Xin Li, Wenping Wang, Chongshou Li, Xueting Li, and Ming-Hsuan Yang. 2025. Controllable 3d outdoor scene generation via scene graphs. arXiv preprint arXiv:2503.07152. — [arXiv:2503.07152](https://arxiv.org/abs/2503.07152) find
  17. Yunze Man, Shihao Wang, Guowen Zhang, Johan Bjorck, Zhiqi Li, Liang-Yan Gui, Jim Fan, Jan Kautz, Yu-Xiong Wang, and Zhiding Yu. 2025. Locateany-thing3d: Vision-language 3d detection with chain-of-sight. arXiv preprint arXiv:2511.20648. — [arXiv:2511.20648](https://arxiv.org/abs/2511.20648) find
  18. Yongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng, Rui Tang, Hao Zhu, Ping Tan, and Zihan Zhou. 2025. Spatiallm: Training large language models for structured indoor modeling. In Advances in Neural Information Processing Systems. find
  19. Yanxu Meng, Haoning Wu, Ya Zhang, and Weidi Xie. 2026. Scenegen: Single-image 3d scene generation in one feedforward pass. In International Conference on 3D Vision 2026. find
  20. Mike Roberts, Jason Ramapuram, Anurag Ranjan, At-ulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. 2021. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision,… find
  21. Katja Schwarz, Denis Rozumny, Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. 2025. A recipe for generating 3d worlds from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3520–3530. find
  22. Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. 2015. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567–576. find
  23. SongTang SongTang, Kaiyong Zhao, Lei Wang, Yuliang Li, Xuebo Liu, Junyi Zou, Qiang Wang, and Xiaowen Chu. 2025. Unrealllm: Towards highly controllable and interactable 3d scene generation by llm-powered procedural content generation. In Findings of the Association for Computational Linguistics: ACL… find
  24. Fan-Yun Sun, Shengguang Wu, Christian Jacobsen, Thomas Yim, Haoming Zou, Alex Zook, Shangru Li, Yu-Hsin Chou, Ethem Can, Xunlei Wu, and 1 others. 2025a. 3d-generalist: Self-improving vision-language-action models for crafting 3d worlds. arXiv preprint arXiv:2507.06484. — [arXiv:2507.06484](https://arxiv.org/abs/2507.06484) find
  25. Weilin Sun, Xinran Li, Manyi Li, Kai Xu, Xiangxu Meng, and Lei Meng. 2025b. Hierarchically-structured open-vocabulary indoor scene synthesis with pre-trained large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7122–7130. find
  26. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, and 1 others. 2025. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense feat…
  27. Xiping Wang, Yuxi Wang, Mengqi Zhou, Junsong Fan, and Zhaoxiang Zhang. 2025a. Hlg: Comprehensive 3d room construction via hierarchical layout generation. arXiv preprint arXiv:2508.17832. — [arXiv:2508.17832](https://arxiv.org/abs/2508.17832) find
  28. Yuxin Wang, Lei Ke, Boqiang Zhang, Tianyuan Qu, Hanxun Yu, Zhenpeng Huang, Meng Yu, Dan Xu, and Dong Yu. 2025b. N3d-vlm: Native 3d grounding enables accurate spatial reasoning in vision-language models. arXiv preprint arXiv:2512.16561. — [arXiv:2512.16561](https://arxiv.org/abs/2512.16561) find
  29. Yuanbo Yang, Yujun Shen, Yue Wang, Andreas Geiger, and Yiyi Liao. 2025c. Urbangen: Urban generation with compositional and controllable neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  30. Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang, Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, and 1 others. 2025d. Mmgdreamer: Mixed-modality graph for geometry-controllable 3d indoor scene generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volum… find
  31. Xingbo Yao, Xuanmin Wang, Hao Wu, Chengliang Ping, Doudou Zhang, and Hui Xiong. 2025a. Magicc-ity: Geometry-aware 3d city generation from satellite imagery with multi-view consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 25325–25334. find
  32. Xingbo Yao, Xuanmin Wang, and Hui Xiong. 2025b. Citysculpt: 3d city generation from satellite imagery with uv diffusion. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 9813– 9821. find
  33. Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. 2025. Wonderworld: Interactive 3d scene generation from a single image. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5916–5926. find
  34. Yunzhi Zhang, Zizhang Li, Matt Zhou, Shangzhe Wu, and Jiajun Wu. 2025. The scene language: Representing scenes with programs, words, and embeddings. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24625–24634. find
  35. Chong Xia, Shengjun Zhang, Fangfu Liu, Chang Liu, Khodchaphun Hirunyaratsameewong, and Yueqi Duan. 2025. Scenepainter: Semantically consistent perpetual 3d scene generation with concept relation alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 28808– 28817… find
  36. Fan Yang, Sicheng Zhao, Yanhao Zhang, Hui Chen, Haonan Lu, Jungong Han, and Guiguang Ding. 2024. Llmi3d: Mllm-based 3d perception from a single 2d image. arXiv preprint arXiv:2408.07422. — [arXiv:2408.07422](https://arxiv.org/abs/2408.07422) find
  37. Yandan Yang, Baoxiong Jia, Shujie Zhang, and Siyuan Huang. 2025a. Sceneweaver: All-in-one 3d scene synthesis with an extensible and self-reflective agent. arXiv preprint arXiv:2509.20414. — [arXiv:2509.20414](https://arxiv.org/abs/2509.20414) find
  38. Yandan Yang, Baoxiong Jia, Shujie Zhang, and Siyuan Huang. 2025b. Sceneweaver: All-in-one 3d scene synthesis with an extensible and self-reflective agent. In Advances in Neural Information Processing Systems (NeurIPS). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2026-scenelm-3d-aware-language-models/