Elamur Research⌕
☰
home / papers / 2026 / See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

Paper · 2026 · Ding Xia, Xinyue Gui, Mark Colley et al.

Summary

Automated vehicles cannot rely on driver gestures, so external Human-Machine Interfaces (eHMIs) such as lightbars, mechanical eyes, or robotic arms must signal intent to pedestrians and other road users. Earlier LLM-based eHMI designers depended on hand-crafted prompts and had no means of checking how their actions actually appear. See2Refine closes this gap with an automated loop that needs no human annotators. A DesignerLLM proposes action sequences for traffic scenarios, Blender renders them as videos seen from the receiver's viewpoint, and a VLM rater scores each video. The scores are turned into preference pairs for iterative DPO fine-tuning. The scenario space is built from six factors, and the action database grows through importance-based sampling and diverse beam search. Across the lightbar, eyes, and arm modalities, the refined designers outperformed prompt-only and manually specified baselines on VLM metrics. A human study confirmed gains over the base model. This matters because it offers a cheap, scalable route to adapting eHMI behaviour, although trust-related judgments remain difficult for VLMs to approximate.

Key points

  • Scenario space: six factors (communication relationship, emitter, receiver, message type, 12 clock directions, three safety distance levels) yield 6,912 combinations.
  • Message generation: GPT-4.1, Claude 3.7 Sonnet, and Qwen3-235B-A22B produced 20,736 intended messages. These were deduplicated with a sentence transformer, and 70% were kept via farthest point sampling.
  • Training pipeline: Qwen2.5 base models (1.5B and 7B Instruct) undergo format-aware fine-tuning, followed by three rounds of DPO driven by VLM preferences. Six 3D eHMI assets (three modalities × car or delivery robot) are used for rendering.
  • Human study setup: 18 participants rated 80 videos (eyes and lightbar, 8 scenario–message pairs, 5 models). The comparison included GPT-5, Claude Opus 4.1, and Grok 4.
  • Human study results: the 7B DesignerLLM improved average scores over its base model by 7.7% (eyes) and 7.9% (lightbar). The largest gains were in targeting (15.4% / 12.6%), and the model ranked 4th overall, behind the commercial LLMs. Code, prompts, and Blender scenes are available at github.com/ApisXia/see2refine.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom107 works · 22,923 citations🎓The University of TokyoTokyo · Japan96 works · 9,600 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksHuman-Computer Interaction · 160 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Ding Xia, Xinyue Gui, Mark Colley, Fan Gao, Zhongyi Zhou, Dongyuan Li, Renhe Jiang and Takeo Igarashi. 2026. See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1044

@inproceedings{xia2026see2refine,
  title = {{See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers}},
  author = {Ding Xia, Xinyue Gui, Mark Colley, Fan Gao, Zhongyi Zhou, Dongyuan Li, Renhe Jiang and Takeo Igarashi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1044},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1044}
}

References

62 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shubham Agrawal, Amy M Schuster, Noah Britt, Elizabeth A Mack, Michael L Tidwell, and Shelia R Cotten. 2023. Building on the past to help prepare the workforce for the future with automated vehicles: A systematic review of automated passenger vehicle deployment timelines. Technology in Society, 72:1… find
  2. Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. 2024. Tuning large multimodal models for videos using reinforcement learning from ai feedback. arXiv preprint arXiv:2402.03746. find
  3. Anthropic. 2025. Claude 3.7 sonnet and claude code. Accessed: 2025-11-02. find
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  6. Pavlo Bazilinskyy, Dimitra Dodou, and Joost De Winter. 2019. Survey on ehmi concepts: The effect of text, color, and perspective. Transportation research part F: traffic psychology and behaviour, 67:175–194. find
  7. Ola Benderius, Christian Berger, and Victor Malmsten Lundgren. 2017. The best rated human–machine interface design for autonomous vehicles in the 2016 grand cooperative driving challenge. IEEE Transactions on intelligent transportation systems, 19(4):1302–1307. find
  8. Blender Foundation. 2025. Home of the blender project — free and open 3d creation software. Accessed: 2025-05-09. find
  9. Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. 2024. On the worst prompt performance of large language models. Advances in Neural Information Processing Systems, 37:69022–69042. find
  10. Chia-Ming Chang, Koki Toda, Xinyue Gui, Stela H Seo, and Takeo Igarashi. 2022. Can eyes on a car reduce traffic accidents? In Proceedings of the 14th international conference on automotive user interfaces and interactive vehicular applications, pages 349–359. find
  11. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-asa-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning. find
  12. Long Chen, Yuchen Li, Chao Huang, Yang Xing, Daxin Tian, Li Li, Zhongxu Hu, Siyu Teng, Chen Lv, Jinjun Wang, and 1 others. 2023. Milestones in autonomous driving and intelligent vehicles—part i: Control, computing system design, communication, hd map, testing, and human behaviors. IEEE Transactions… find
  13. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  14. Mark Colley, Pascal Jansen, Mugdha Keskar, and Enrico Rukzio. 2025. Improving external communication of automated vehicles using bayesian optimization. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA. Association for Computing Machinery. find
  15. Mark Colley and Enrico Rukzio. 2020. A design space for external communication of autonomous vehicles. In 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pages 212–222. find
  16. Mark Colley, Marcel Walch, Jan Gugenheimer, Ali Askari, and Enrico Rukzio. 2020. Towards inclusive external communication of autonomous vehicles for pedestrians with vision impairments. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–14. find
  17. Joost de Winter and Dimitra Dodou. 2022. External human–machine interfaces: Gimmick or necessity? Transportation research interdisciplinary perspectives, 15:100643. find
  18. Shuchisnigdha Deb, Lesley J Strawderman, and Daniel W Carruth. 2018. Investigating pedestrian suggestions for external features on fully autonomous vehicles: A virtual reality experiment. Transportation research part F: traffic psychology and behaviour, 59:135–149. find
  19. Debargha Dey, Azra Habibovic, Andreas Löcken, Philipp Wintersberger, Bastian Pfleging, Andreas Riener, Marieke Martens, and Jacques Terken. 2020a. Taming the ehmi jungle: A classification taxonomy to guide, compare, and assess the design principles of automated vehicles’ external human-machine inter… find
  20. Debargha Dey, Azra Habibovic, Bastian Pfleging, Marieke Martens, and Jacques Terken. 2020b. Color and animation preferences for a light band ehmi in interactions between automated vehicles and pedestrians. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–13. find
  21. Yuhao Du, Shunian Chen, Wenbo Zan, Peizhao Li, Mingxuan Wang, Dingjie Song, Bo Li, Yan Hu, and Benyou Wang. 2024. Blenderllm: Training large language models for computer-aided design with self-improvement. Preprint, arXiv:2412.14203. — [arXiv:2412.14203](https://arxiv.org/abs/2412.14203) find
  22. Daniel Eisele and Tibor Petzoldt. 2022. Effects of traffic context on ehmi icon comprehension. Transportation research part F: traffic psychology and behaviour, 85:1–12. find
  23. Yke Bauke Eisma, Anna Reiff, Lars Kooijman, Dimitra Dodou, and Joost CF de Winter. 2021. External human-machine interfaces: Effects of message perspective. Transportation research part F: traffic psychology and behaviour, 78:30–41. find
  24. Yuval Eldar, Michael Lindenbaum, Moshe Porat, and Yehoshua Y Zeevi. 1997. The farthest point strategy for progressive image sampling. IEEE transactions on image processing, 6(9):1305–1315. find
  25. Daniel J Fagnant and Kara Kockelman. 2015. Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice, 77:167–181. find
  26. Xinyue Gui, Chia-Ming Chang, Stela H Seo, Koki Toda, and Takeo Igarashi. 2024a. Scenarios exploration: How ar-based speech balloons enhance car-to-pedestrian interaction. In International Conference on Human-Computer Interaction, pages 223–230. Springer. find
  27. Xinyue Gui, Mikiya Kusunoki, Bofei Huang, Stela Hanbyeol Seo, Chia-Ming Chang, Haoran Xie, Manabu Tsukada, and Takeo Igarashi. 2024b. Shrinkable arm-based ehmi on autonomous delivery vehicle for effective communication with other road users. In Proceedings of the 16th International Conference on Aut… find
  28. Xinyue Gui, Koki Toda, Stela Hanbyeol Seo, Chia-Ming Chang, and Takeo Igarashi. 2022. “i am going this way”: Gazing eyes on self-driving car show multiple driving directions. In Proceedings of the 14th international conference on automotive user interfaces and interactive vehicular applications, pag… find
  29. Xinyue Gui, Koki Toda, Stela Hanbyeol Seo, Felix Martin Eckert, Chia-Ming Chang, Xiang’Anthony Chen, and Takeo Igarashi. 2023. A field study on pedestrians’ thoughts toward a car with gazing eyes. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–7. find
  30. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  31. Hugging Face. 2024. Number tokenization blog. https://huggingface.co/spaces/ huggingface/number-tokenization-blog. ↗
  32. Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, and Mahdi Namazifar. 2023. Data-efficient alignment of large language models with human feedback through natural language. arXiv preprint arXiv:2311.14543. — [arXiv:2311.14543](https://arxiv.org/abs/2311.14543) find
  33. Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. 2023. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925.
  34. Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2):155–163. find
  35. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and 1 others. 2023. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. find
  36. Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. 2024a. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. In Findings of the association for computational linguistics ACL 2024, pages 11286–11315. find
  37. Tony Lee, Haoqin Tu, Chi H Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin S Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, and 1 others. 2024b. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems, 37:140632–140666. find
  38. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  39. Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. 2024. Wildvision: Evaluating vision-language models in the wild with human preferences. Advances in Neural Information Processing Systems, 37:48224–48255. find
  40. Karthik Mahadevan, Sowmya Somanath, and Ehud Sharlin. 2018. Communicating awareness and intent in autonomous vehicle-pedestrian interaction. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1–12. find
  41. Subir Majumder, Lin Dong, Fatemeh Doudi, Yuting Cai, Chao Tian, Dileep Kalathil, Kevin Ding, Anupam A Thatte, Na Li, and Le Xie. 2024. Exploring the capabilities and limitations of large language models in the electric energy sector. Joule, 8(6):1544–1549. find
  42. H. Nyquist. 1928. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2):617–644. find
  43. Yoichi Ochiai and Keisuke Toyoshima. 2011. Homuncu-lus: the vehicle as augmented clothes. In Proceedings of the 2nd Augmented Human International Conference, pages 1–4. find
  44. OpenAI. 2025. Introducing gpt-4.1 in the api. Accessed: 2025-11-02. find
  45. Qwen Team. 2024. Qwen2.5 technical report. https://qwenlm.github.io/blog/qwen2.5/. Accessed: 2025-11-02. ↗
  46. Qwen Team, Alibaba Cloud. 2025. Qwen3-235b-a22b: Hybrid reasoning MoE model. Accessed: 2025-11-02. find
  47. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  48. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  49. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  50. Alexandros Rouchitsas and Håkan Alm. 2019. External human–machine interfaces for autonomous vehicle-to-pedestrian communication: A review of empirical work. Frontiers in psychology, 10:2757. find
  51. Dimitris Spathis and Fahim Kawsar. 2023. The first step is the hardest: Pitfalls of representing and tokenizing temporal data for large language models. arXiv preprint arXiv:2309.06236. — [arXiv:2309.06236](https://arxiv.org/abs/2309.06236) find
  52. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, and 1 others. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525.
  53. Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. 2020. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):25–55. find
  54. Binghai Wang, Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, and 1 others. 2025. Worldpm: Scaling human preference modeling. arXiv preprint arXiv:2505.10527. find
  55. Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. 2024. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. In Proceedings of the 41th International Conference on Machine Learning. find
  56. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  57. Ding Xia, Xinyue Gui, Fan Gao, Dongyuan Li, Mark Colley, and Takeo Igarashi. 2025. Automating ehmi action design with llms for automated vehicle communication. arXiv preprint arXiv:2505.20711. find
  58. Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. 2025. Llava-critic: Learning to evaluate multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13618–13628. find
  59. Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, and 1 others. 2025. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19985–19995. find
  60. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), pages 400–410.
  61. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  62. Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2026-see2refine-vision-language-feedback-improves/