Elamur Research⌕
☰
home / papers / 2026 / A Virtual Assistant for Architectural Design in a VR Environment

A Virtual Assistant for Architectural Design in a VR Environment

Paper · 2026 · Ander Salaberria, Oier Ijurco, Markel Ferro et al.

Summary

Editing Building Information Modeling (BIM) files usually requires expert software skills. This slows down iterative architectural prototyping and shuts out non-specialists. The authors present VR-ARCH, a voice-driven assistant that lets users inspect and change a BIM-based building while immersed in virtual reality.

Spoken requests are transcribed and passed to a ReAct-style Router LLM. At each step, the Router picks one of three prompt-engineered tools:

  • a Modifier, which writes Python code against a custom Unreal Engine sandbox API;
  • a Querier, which writes Cypher queries over BIM data stored in Neo4j;
  • an ID Retriever, which locates objects using spatial cues relative to the user's position.

Chaining these tools supports composite requests, such as asking for the material of a particular wall. Replies are spoken back via text-to-speech. Evaluation with human judges and LLM-as-a-judge shows that LLMs can answer questions and carry out modifications reliably, with quality rising as model size grows. Hallucinations and inefficient routing remain open problems. Code and data are public, and the setup also doubles as a benchmark for agentic behaviour in realistic settings.

Key points

  • The pipeline has six components: speech-to-text, the ReAct Router, the Unreal Engine sandbox with its Python API, the Neo4j graph database (Cypher), the three tools (Modifier, Querier, ID Retriever) and text-to-speech.
  • The sandbox API can recolor, move, hide or reveal objects, control the camera, and add, transform or delete props. Spatial commands take the user's position into account.
  • Each LLM is adapted only through prompting, using task definitions, API docs or graph schemas, and few-shot examples; there is no fine-tuning.
  • In the step analysis, most correct answers use the optimal number of steps. One extra step can still succeed, but accuracy falls sharply with longer trajectories.
  • In a case study, a 32B model recovers from a wrong first tool choice, whereas a 4B model loops without finishing. Stated limitations are that extending the sandbox API takes manual work, the system has no visual grounding, and results depend on BIM file quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of the Basque CountryLeioa · Spain57 works · 16,520 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksEngineering · 133 worksHuman-Computer Interaction · 141 worksArchitecture · 2 works

Cite

Cite as Ander Salaberria, Oier Ijurco, Markel Ferro, Jiayuan Wang, Iñigo Vilá Muñoz, Roberto de Ioris, Jeremy Barnes and Oier Lopez De Lacalle. 2026. A Virtual Assistant for Architectural Design in a VR Environment. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.38

@inproceedings{salaberria2026virtual,
  title = {{A Virtual Assistant for Architectural Design in a VR Environment}},
  author = {Ander Salaberria, Oier Ijurco, Markel Ferro, Jiayuan Wang, Iñigo Vilá Muñoz, Roberto {de Ioris}, Jeremy Barnes and Oier Lopez {De Lacalle}},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.38},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.38}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maia Aguirre, Ariane Méndez, Aitor García-Pablos, Montse Cuadros, Arantza del Pozo, Oier Lopez de Lacalle, Ander Salaberria, Jeremy Barnes, Pablo Martin Fischer. 2006. Formalizing construction knowledge for concurrent performance-based design. In Workshop of the European group for intelligent comput… find
  2. Silvia García-Méndez, Francisco de Arriba-Pérez, and María del Carmen Somoza-López. 2024. A review on the use of large language models as virtual tutors. Science & Education, pages 1–16. find
  3. Daniele Giunchi, Nels Numan, Elia Gatti, and Anthony Steed. 2024. Dreamcodevr: Towards democratizing behavior design in virtual reality with speech-driven programming. 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 579–589. find
  4. Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962. find
  5. International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives, pages 85–90, Marseille, France. European Language Resources Association. find
  6. Suhyung Jang and Ghang Lee. 2024. Interactive design by integrating a large pre-trained language model and building information modeling. In Computing in Civil Engineering 2023, Computing in Civil Engineering 2023: Visualization, Information Modeling, and Simulation - Selected Papers from the ASCE I… find
  7. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. find
  8. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. Preprint, arXiv:2405.01535.
  9. Mikhail Konenkov, Artem Lykov, Daria Trinitatova, and Dzmitry Tsetserukou. 2024. VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications. Preprint, arXiv:2405.11537. — [arXiv:2405.11537](https://arxiv.org/abs/2405.11537) find
  10. Fernanda L Leite. 2019. BIM for design coordination: A virtual design and construction guide for designers, general contractors, and MEP subcontractors. John Wiley & Sons. find
  11. Ang Li, Peter Kok-Yiu Wong, Xingyu Tao, Jun Ma, and Jack C.P. Cheng. 2025. An interactive system for 3d spatial relationship query by integrating tree-based element indexing and llm-based agent. Advanced Engineering Informatics, 66:103375. find
  12. Makbule Gulcin Ozsoy, Leila Messallem, Jon Besga, and Gianandrea Minneci. 2025. Text2cypher: Bridging natural language and graph databases. In Proceedings of the Workshop on Generative AI and Knowledge Graphs (GenAIK), pages 100–108. find
  13. Alexander Prange, Margarita Chikobava, Peter Poller, Michael Barz, and Daniel Sonntag. 2017. A multimodal dialogue system for medical decision support inside virtual reality. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 23–26, Saarbrücken, Germany. Association f… find
  14. Lorna Quandt, Jason Lamberton, Carly Leannah, Athena Willis, and Melissa Malzkuhn. 2022. Signing avatars in a new dimension: Challenges and opportunities in virtual reality. In Proceedings of the 7th Rafael Sacks, Tanya Bloch, Meir Katz, and Raz Yosef. 2019. Automating Design Review with Artificial… find
  15. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. In Advances in Neural Information Processing Systems, volume 36, pages 38154–38180. Curran Associates, Inc.
  16. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. Preprint, arXiv:2408.03314.
  17. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11888–11898.
  18. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  19. Monica Ward, Liang Xu, and Elaine Uí Dhonnchadha. 2025. A pragmatic approach to using artificial intelligence and virtual reality in digital game-based language learning. In Proceedings of the 5th Celtic Language Technology Workshop, pages 27–34, Abu Dhabi [Virtual Workshop]. International Committee…
  20. Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, and Seungwhan Moon. 2023. SIMMC-VR: A task-oriented multimodal dialog dataset with situated and immersive VR streams. In Proceedings of the 61st Annual Meeting of the Association for Computational… find
  21. Takato Yamazaki, Tomoya Mizumoto, Katsumasa Yoshikawa, Masaya Ohagi, Toshiki Kawamoto, and Toshinori Sato. 2023. An open-domain avatar chatbot by exploiting a large language model. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 428–432, Pragu… find
  22. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  23. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. Mm-react: Prompting chatgpt for multimodal reasoning and action. Preprint, arXiv:2303.11381. find
  24. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  25. Junwen Zheng and Martin Fischer. 2023. Dynamic prompt-based virtual assistant framework for bim information search. Automation in Construction, 155:105067. find
  26. Junxiang Zhu, Nicholas Nisbet, Mengtian Yin, Ran Wei, and Ioannis Brilakis. 2024. Cypher4bim: Releasing the power of graph for building knowledge discovery. arXiv preprint arXiv:2405.16345. *Figure 5: Outside of the School BIM environment visualized in the custom Unreal Engine Sandbox.* *Figure 7: O… — [arXiv:2405.16345](https://arxiv.org/abs/2405.16345) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salaberria-2026-virtual-assistant-architectural-design-vr/