Elamur Research⌕
☰
home / papers / 2026 / Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs

Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs

Paper · 2026 · Sandeep Mishra, Devichand Budagam, Anubhab Mandal et al.

Summary

The paper addresses inline auto-completion in chats where users share images, so the right continuation of a half-typed message may depend on what a picture shows. The authors define this as Multimodal Auto-Completion (MAC), an extension of text-only auto-completion (TAC). To build benchmarks, they take MMDialog and ImageChat, use GPT-4V to keep only single-image dialogs where the image matters, and evaluate two groups of models. The first group is text-only systems: MPC, MPC++ and QueryBlazer. The second is vision-language models: MiniCPM-V, PaliGemma and Qwen2-VL. VLMs give better completions but are far slower.

To address this, they propose Router-Suggest, a small neural classifier that chooses, for each prefix, which model should answer. It is trained on EmbeddingGemma-300m features with a cost-aware objective. Router-Suggest matches or beats the strongest VLM in partial-F1 while running 2.3–10× faster. A user study found that VLM-based suggestions save more typing and earn higher satisfaction. The work makes a case for latency-aware multimodal typing assistants. Code and benchmarks are public.

Key points

  • Benchmark sizes:
    • MMDD (adapted MMDialog) has 13,182 train and 893 test dialogs, averaging about 12 utterances each.
    • ImageChat has 186,724 train and 9,994 test dialogs, with only 2–3 utterances each.
  • Router training:
    • Gold labels come from whichever model scores the best partial-F1 on each sample.
    • Inputs are 768-dimensional EmbeddingGemma-300m prefix embeddings.
    • The loss combines cross-entropy with an expected-latency cost term.
  • MMDD results:
    • MiniCPM-V reaches partial-F1 0.247 at 2.08 s.
    • Router-4-P reaches 0.281 at 0.832 s.
    • Router-2-L reaches 0.240 at 0.170 s.
    • An oracle router would reach 0.356, showing room for improvement.
  • User study: 190 sessions compared MPC, QB, MiniCPM-V and Router-2-L. MiniCPM-V achieved the highest Typing Effort Saved (TES) and ratings, Router-2-L scored similarly, and TES tracked user ratings closely.
  • Limitations: GPT-4V filtering may bias the data toward visually explicit cases. Only single-image contexts are covered. The router may degrade under domain shift.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft (India)Hyderabad · India10 works · 2,129 citations🎓Indian Institute of Technology KharagpurKharagpur · India24 works · 714 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksVision & language · 657 works

Cite

Cite as Sandeep Mishra, Devichand Budagam, Anubhab Mandal, Bishal Santra, Pawan Goyal and Manish Gupta. 2026. Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.11

@inproceedings{mishra2026router,
  title = {{Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs}},
  author = {Sandeep Mishra, Devichand Budagam, Anubhab Mandal, Bishal Santra, Pawan Goyal and Manish Gupta},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.11},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.11}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lars Backstrom, Jon Kleinberg, Ravi Kumar, and Jasmine Novak. 2008. Spatial variation in search engine queries. In Proceedings of the 17th international conference on World Wide Web, pages 357– 366. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  3. Ziv Bar-Yossef and Naama Kraus. 2011. Context-sensitive query auto-completion. In Proceedings of the 20th international conference on World wide web, pages 107–116. find
  4. Holger Bast and Ingmar Weber. 2006. Type less, find more: fast autocompletion search with a succinct index. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 364–371. find
  5. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Ke… find
  6. Rohan Chitnis, Shentao Yang, and Alborz Geramifard. 2024. Sequential decision-making for inline text autocomplete. arXiv preprint arXiv:2403.15502. — [arXiv:2403.15502](https://arxiv.org/abs/2403.15502) find
  7. Giovanni Di Santo, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2015. Comparing approaches for query autocompletion. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 775–778. find
  8. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and 1 others. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192…
  9. Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. 2023. MMDialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (… find
  10. Young Mo Kang, Wenhao Liu, and Yingbo Zhou. 2021. Queryblazer: Efficient query autocompletion framework. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 1020–1028. find
  11. Gyuwan Kim. 2019. Subword language model for query auto-completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5022–5032, Hong Kong, China. Association for C… find
  12. Fanheng Kong, Peidong Wang, Shi Feng, Daling Wang, and Yifei Zhang. 2024. TIGER: A unified generative model framework for multimodal dialogue response generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 20… find
  13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  14. Dong-Ho Lee, Zhiqiang Hu, and Roy Ka-Wei Lee. 2021. Improving text auto-completion with next phrase prediction. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4434–4438, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  15. Bolun Li, Zhihong Sun, Tao Huang, Hongyu Zhang, Yao Wan, Ge Li, Zhi Jin, and Chen Lyu. 2024. Ircoco: Immediate rewards-guided deep reinforcement learning for code completion. Proceedings of the ACM on Software Engineering, 1(FSE):182– 203. find
  16. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. Workshop on Text Summariation Branches Out, Post-Conference Workshop of ACL 2004, pages 74–81.
  17. Bo Liu, Lejian He, Yafei Liu, Tianyao Yu, Yuejia Xiang, Li Zhu, and Weijian Ruan. 2022. Transformer-based multimodal infusion dialogue systems. Electronics, 11(20). find
  18. Sandeep Mishra, Anubhab Mandal, Bishal Santra, Tushar Abhishek, Pawan Goyal, and Manish Gupta. 2025. Chat-ghosting: A comparative study of methods for auto-completion in dialog systems. Preprint, arXiv:2507.05940. — [arXiv:2507.05940](https://arxiv.org/abs/2507.05940) find
  19. Agnès Mustar, Sylvain Lamprier, and Benjamin Piwowarski. 2020. Using bert and bart for query suggestion. In Joint Conference of the Information Retrieval Communities in Europe, volume 2621. CEUR-WS. org. find
  20. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/ chatgpt. Accessed July 2025. ↗
  21. OpenAI. 2023. Gpt-4v(ision) technical work and authors. https://openai.com/ contributions/gpt-4v/. ↗
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  23. Lakshmi Ramachandran and Uma Murthy. 2019. Ghosting: contextualized query auto-completion on amazon search. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pages 1377– 1378. find
  24. Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2020. Image-chat: Engaging grounded conversations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2414–2429, Online. Association for Computational Linguistics. find
  25. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In proceedings of the 24th ACM international on conference on information and knowledge management,… find
  26. Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yaming Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, and Daxin Jiang. 2022. Multimodal dialogue response generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2854–2866,… find
  27. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  28. Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, and 70 others. 2025. Embedd… — [arXiv:2509.20354](https://arxiv.org/abs/2509.20354) find
  29. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  30. Sida Wang, Weiwei Guo, Huiji Gao, and Bo Long. 2020. Efficient neural query auto completion. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2797–2804. find
  31. Stewart Whiting, Andrew James McMinn, and Joemon M Jose. 2013. Exploring real-time temporal query auto-completion. In DIR, pages 12–15. find
  32. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. Qwen2 technical report. Preprint, arXiv:240… find
  33. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
  34. Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Kang Zhang, Yu-Jung Heo, Du-Seong Chang, and Chang D Yoo. 2024. Bi-mdrg: Bridging image history in multimodal dialogue response generation. In European Conference on Computer Vision, pages 378–396. Springer. find
  35. Prompt Template You are a discriminator model who will decide if the following hold: \#\# 1. The dialog is relevant to the image. find
  36. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mishra-2026-router-suggest-dynamic-routing-multimodal/