Elamur Research⌕
☰
home / papers / 2026 / Context-Aware Language Understanding in Human-Robot Dialogue with LLMs

Context-Aware Language Understanding in Human-Robot Dialogue with LLMs

Paper · 2026 · Svetlana Stoyanchev, Youmna Farag, Simon Keizer et al.

Summary

The paper examines whether off-the-shelf LLMs, without fine-tuning, can turn a user's spoken-style instructions to a household robot into executable sequences of API calls. The focus is on the dynamic case where a person cuts in while the robot is still acting, for example to correct it or help it recover from a failed step. The authors built a text-and-video interface to a virtual robot in the AI2THOR simulator, extended by TEACh. Using an expert-in-the-loop procedure, an expert guided the robot through tasks and supplied correct interpretations whenever the automatic ones failed. This produced a dataset of 484 annotated utterances, which the authors describe as the first to include directives issued mid-execution. They compared gpt-5-mini and Qwen3-4B using leave-one-out evaluation and few-shot examples retrieved by a weighted similarity over objects, previous plans, utterances and context type. Similarity-based retrieval clearly outperformed random selection. However, interruptions and failure-recovery utterances stayed difficult for both models, which points to the need for context-adaptive interpretation strategies.

Key points

  • The dataset contains 484 utterances, including plan-interrupting and failure-correcting directives, and is released on GitHub (sstoyanchev/Human-directing-robot-dialogue).
  • With 5 similarity-selected examples, gpt-5-mini reached 67% exact-match accuracy and Qwen3-4B reached 59%. Both scores beat prompting with 40 randomly chosen examples.
  • Examples are retrieved with all-MiniLM-L6-v2 similarity, combining objects, plans and utterances with equal weights (α=β=γ=1). Context type gets a large weight (δ=100) so that retrieved examples match the current situation.
  • For directives given during execution, accuracy was at most 0.5. Qwen fell well behind GPT after execution failures (e.g., 0.303 vs 0.424 with 10 examples).
  • Qwen is more sensitive to the number of examples. On initial commands, cutting from 10 to 3 examples dropped Qwen from 0.616 to 0.411, while GPT only fell from 0.575 to 0.507. Adding CONTINUE shortcuts raised accuracy on interruptions after successful execution from about 0.21–0.24 to 0.500.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Svetlana Stoyanchev, Youmna Farag, Simon Keizer, Mohan Li and Rama Sanand Doddipatla. 2026. Context-Aware Language Understanding in Human-Robot Dialogue with LLMs. Proceedings of the 16th International Workshop on Spoken Dialogue System Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{stoyanchev2026context,
  title = {{Context-Aware Language Understanding in Human-Robot Dialogue with LLMs}},
  author = {Svetlana Stoyanchev, Youmna Farag, Simon Keizer, Mohan Li and Rama Sanand Doddipatla},
  booktitle = {Proceedings of the 16th International Workshop on Spoken Dialogue System Technology},
  year = {2026},
  url = {https://aclanthology.org/2026.iwsds-1.27/}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Adiga, Lakshminarayanan Subramanian, and Varun Chandrasekaran. 2024. Designing informative metrics for few-shot example selection. In Annual Meeting of the Association for Computational Linguistics. find
  2. Michael Ahn et al. 2022. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691.
  3. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2017. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. 2018 IEEE/CVF Conference on Computer Vision and Pat…
  4. Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022. A persistent spatial semantic representation for high-level natural language instruction execution. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 706–71… find
  5. PMLR. Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-tur. 2020. Just ask: An interactive learning framework for vision and language navigation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(03):2459–2466. find
  6. Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. 2024. Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks. In The Twelvth International Conference on Learning Representations ICLR 2024. find
  7. Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  8. Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S. Sukhatme. 2022. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 7(4):10049–10056. find
  9. Gemini Robotics Team et al. 2025. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer. find
  10. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118–9147.
  11. PMLR. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022b. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. find
  12. Chenxi Jiang, Chuhao Zhou, and Jianfei Yang. 2025. REI-bench: Can embodied agents understand vague human instructions in task planning? Preprint, arXiv:2505.10872. — [arXiv:2505.10872](https://arxiv.org/abs/2505.10872) find
  13. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025. OpenVLA: An Open-Source Vision-… find
  14. PMLR. Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. 2022. Ai2-thor: An interactive 3d environment for visual ai. Preprint, arXiv:1712.05474. find
  15. Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. 2023. Text2motion: From natural language instructions to feasible plans. In ICRA2023 Workshop on Pretraining for Robotics (PT4R). find
  16. So Yeon Min, Hao Zhu, Ruslan Salakhutdinov, and Yonatan Bisk. 2022. Don’t copy the teacher: Data and model challenges in embodied dialogue. Preprint, arXiv:2210.04443. find
  17. Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. 2019. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12519–12529. Aishwarya Padmakumar, Jesse… find
  18. Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022b. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages… find
  19. Branislav Pecher, Ivan Srba, Maria Bielikova, and Joaquin Vanschoren. 2024. Automatic combination of sample selection strategies for few-shot learning. Preprint, arXiv:2402.03038.
  20. Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. 2023. Open-ended instructable embodied agents with memory-augmented large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. Ying Shen, Daniel Biś, Cynthia Lu, and Ismini Lourentzou. 2025. Elba: L… find
  21. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  22. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1…
  23. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  24. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 394–406. PMLR. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian (Shawn) M… find
  25. An Yang et al. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  26. Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, and Joyce Chai. 2022. DANLI: Deliberative agent for following natural language instructions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1280–1298… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stoyanchev-2026-context-aware-language-understanding-human/