Elamur Research⌕
☰
home / papers / 2026 / CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering

CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering

Paper · 2026 · Xiyin Zeng, Yi Lu and Hao Wang

Summary

Mixture-of-Experts (MoE) vision–language models for multiple-choice VQA face a routing trade-off. If routing is unstable, differently phrased questions of the same type get sent to different experts. If routing is made very stable, as in MoKE or ER-MoE, the activated expert set freezes and the model becomes worse at telling candidate answers apart.

CoGR-MoE addresses both problems in three steps:

  • Cue generation. An LLM produces positive and negative visual cues for every answer option. An agreement/variance uncertainty score triggers regeneration of cues that align poorly with the image.
  • Concept-guided routing. During training, a semantic direction built from the correct option's cues is added to the router logits. This teacher signal is then distilled into a student router, giving consistent Top-K expert selection.
  • Option-aware reweighting. Each option's cues adjust how much the selected experts contribute, producing a separate representation per option. These representations are trained with uncertainty-weighted cross-entropy and a contrastive loss.

Built on MoE-LLaVA, the method beats strong MoE baselines on several VQA benchmarks, with accuracy gains of up to 4.9 points. It also makes expert usage easier to interpret. This shows that stable expert selection and flexible expert weighting can be combined rather than traded off.

Key points

  • Option uncertainty is defined as unc = Var / (1 + Agr). Var measures how much cue–image similarity fluctuates across paraphrased cues, and Agr measures how well the cues agree with the image. The semantic direction is added to the router logits with a fixed strength λ_a = 0.5.
  • Evaluation covers VMCBench (including its VQAv2 and ScienceQA subsets), MRAG-Bench, GQA, VizWiz, ScienceQA, MMVet and MMStar. Reported gains reach up to 4.9% absolute accuracy.
  • In ablations, removing any component lowers results on nearly all benchmarks; the only exception is that dropping unc_j slightly helps on MMVet, which the authors attribute to uncertainty weighting down-weighting cue-weak examples. Prompt-only cue use, which leaves the router untrained, and removing the routing direction s_a cause the largest drops.
  • On VMCBench, the best configuration uses n = 8 experts (71.6 accuracy with K = 1) and K = 2 active experts (74.5 accuracy). Larger K values dilute semantic alignment.
  • Stated limitations:
    • The method depends on external LLM-generated cues, which adds training cost and exposes it to hallucinations.
    • It underperforms interaction-driven routers such as I²MoE on some interaction-heavy subtasks.
    • Evaluation is limited to multiple-choice VQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China53 works · 1,367 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 worksQuestion answering · 441 works

Cite

Cite as Xiyin Zeng, Yi Lu and Hao Wang. 2026. CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.315

@inproceedings{zeng2026cogr,
  title = {{CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering}},
  author = {Xiyin Zeng, Yi Lu and Hao Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.315},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.315}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ekaterina Borisova, Nikolas Rauscher, and Georg Rehm. 2025. find
  2. SciVQA 2025: Overview of the first scientific visual question answering shared task. In Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025), pages 182–210, Vienna, Austria. Association for Computational Linguistics. find
  3. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, page 1–20. find
  4. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024. Are we on the right way for evaluating large vision-language models? Preprint, arXiv:2403.20330.
  5. Long Chen, Yuhang Zheng, Yulei Niu, Hanwang Zhang, and Jun Xiao. 2023. Counterfactual samples synthesizing and training for robust visual question answering. IEEE Trans. Pattern Anal. Mach. Intell., 45(11):13218–13234. find
  6. Anzhe Cheng, Shukai Duan, Shixuan Li, Chenzhong Yin, Mingxi Cheng, Heng Ping, Tamoghna Chattopadhyay, Sophia I Thomopoulos, Shahin Nazarian, Paul Thompson, and Paul Bogdan. 2025a. Ermoe: Eigen-reparameterized mixture-of-experts for stable routing and interpretable specialization. Preprint, arXiv:251… — [arXiv:2511.10971](https://arxiv.org/abs/2511.10971) find
  7. YuJu Cheng, Yu-Chu Yu, Kai-Po Chang, and Yu-Chiang Frank Wang. 2025b. Serial lifelong editing via mixture of knowledge experts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30888–30903, Vienna, Austria. Association for Comp… find
  8. Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2025. Mxmoe: Mixed-precision quantization for moe with accuracy and performance co-design. Preprint, arXiv:2505.05799. — [arXiv:2505.05799](https://arxiv.org/abs/2505.05799) find
  9. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. Preprint, arXiv:1612.00837.
  10. Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. Preprint, arXiv:1802.08218. — [arXiv:1802.08218](https://arxiv.org/abs/1802.08218) find
  11. Xing Han, Hsing-Huan Chung, Joydeep Ghosh, Paul Pu Liang, and Suchi Saria. 2025. Guiding mixture-of-experts with temporal multimodal interactions. Preprint, arXiv:2509.25678. — [arXiv:2509.25678](https://arxiv.org/abs/2509.25678) find
  12. Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. 2024. Mrag-bench: Vision-centric evaluation for retrieval-augmented multimodal models. arXiv preprint arXiv:2410.08182. — [arXiv:2410.08182](https://arxiv.org/abs/2410.08182) find
  13. Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. 2025. Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering. arXiv preprint arXiv:2503.00413. — [arXiv:2503.00413](https://arxiv.org/abs/2503.00413) find
  14. Shaohan Huang, Xun Wu, Shuming Ma, and Furu Wei. 2024. Mh-moe: Multi-head mixture-of-experts. Preprint, arXiv:2411.16205. — [arXiv:2411.16205](https://arxiv.org/abs/2411.16205) find
  15. Siyuan Mu and Sen Lin. 2025. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. Preprint, arXiv:2503.07137. — [arXiv:2503.07137](https://arxiv.org/abs/2503.07137) find
  16. Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Preprint, arXiv:1902.09506.
  17. Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, and Jun Suzuki. 2025. Drop-upcycling: Training sparse mixture of experts with partial re-initialization. In The Thirteenth International Conference on Learning Representations. find
  18. Byeong Su Kim, Jieun Kim, Deokwoo Lee, and Beakcheol Jang. 2025. Visual question answering: A survey of methods, datasets, evaluation, and challenges. ACM Comput. Surv., 57(10). find
  19. Xiaohan Lan, Fanfan Liu, Haibo Qiu, Siqi Yang, Delian Ruan, Peng Shi, and Lin Ma. 2025. Metis-home: Hybrid optimized mixture-of-experts for multimodal reasoning. arXiv preprint arXiv:2510.20519. — [arXiv:2510.20519](https://arxiv.org/abs/2510.20519) find
  20. Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. 2025. Mixture of experts meets prompt-based continual learning. Preprint, arXiv:2405.14124. — [arXiv:2405.14124](https://arxiv.org/abs/2405.14124) find
  21. Jing Li, Zhijie Sun, Dachao Lin, Xuan He, Binfan Zheng, Yi Lin, Rongqian Zhao, and Xin Chen. 2025a. Expert-token resonance moe: Bidirectional routing with efficiency affinity-driven active selection. Preprint, arXiv:2406.00023. — [arXiv:2406.00023](https://arxiv.org/abs/2406.00023) find
  22. Yunxin Li, Xinyu Chen, Shenyuan Jiang, Haoyuan Shi, Zhenyu Liu, Xuanyu Zhang, Nanhao Deng, Zhenran Xu, Yicheng Ma, Meishan Zhang, Baotian Hu, and Min Zhang. 2025b. Uni-moe-2.0-omni: Scaling language-centric omnimodal large model with advanced moe, training and data. Preprint, arXiv:2511.12609. — [arXiv:2511.12609](https://arxiv.org/abs/2511.12609) find
  23. Zhongyang Li, Ziyue Li, and Tianyi Zhou. 2025c. R2-t2: Re-routing in test-time for multimodal mixture-of-experts. Preprint, arXiv:2502.20395. — [arXiv:2502.20395](https://arxiv.org/abs/2502.20395) find
  24. Zhongyang Li, Ziyue Li, and Tianyi Zhou. 2025d. Routing manifold alignment improves generalization of mixture-of-experts llms. Preprint, arXiv:2511.07419. — [arXiv:2511.07419](https://arxiv.org/abs/2511.07419) find
  25. Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947. — [arXiv:2401.15947](https://arxiv.org/abs/2401.15947) find
  26. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. Preprint, arXiv:2310.03744.
  27. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022a. Learn to explain: Multimodal reasoning via thought chains for science question answering. Preprint, arXiv:2209.09513. find
  28. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2022b. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. Preprint, arXiv:2110.13214. find
  29. Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, and Vasudev Lal. 2025. Probing semantic routing in large mixture-of-expert models. Preprint, arXiv:2502.10928. find
  30. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. Preprint, arXiv:2206.01718.
  31. Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. 2024a. Mome: Mixture of multimodal experts for generalist multimodal large language models. Preprint, arXiv:2407.12709. — [arXiv:2407.12709](https://arxiv.org/abs/2407.12709) find
  32. Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. 2024b. Mome: Mixture of multimodal experts for generalist multimodal large language models. In Advances in Neural Information Processing Systems (NeurIPS). find
  33. Vinay Kumar Verma, Shreyas Sunil Kulkarni, Happy Mittal, and Deepak Gupta. 2025. Moemoe: Question guided dense and scalable sparse mixture-of-expert for multi-source multi-modal answering. Preprint, arXiv:2503.06296. find
  34. Yujie Wei, Shiwei Zhang, Hangjie Yuan, Yujin Han, Zhekai Chen, Jiayu Wang, Difan Zou, Xihui Liu, Yingya Zhang, Yu Liu, and Hongming Shan. 2025. Routing matters in moe: Scaling diffusion transformers with explicit routing guidance. Preprint, arXiv:2510.24711. — [arXiv:2510.24711](https://arxiv.org/abs/2510.24711) find
  35. Zhiquan Wen, Yaowei Wang, Mingkui Tan, Qingyao Wu, and Qi Wu. 2023. Digging out discrimination information from generated samples for robust visual question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6910–6928, Toronto, Canada. Association for Computatio… find
  36. Qiong Wu, Zhaoxi Ke, Yiyi Zhou, Xiaoshuai Sun, and Rongrong Ji. 2025. Routing experts: Learning to route dynamic experts in multi-modal large language models. Preprint, arXiv:2407.14093. — [arXiv:2407.14093](https://arxiv.org/abs/2407.14093) find
  37. Jiayi Xin, Sukwon Yun, Jie Peng, Inyoung Choi, Jenna L Ballard, Tianlong Chen, and Qi Long. 2025. I2moe: Interpretable multimodal interaction-aware mixture-of-experts. arXiv preprint arXiv:2505.19190. — [arXiv:2505.19190](https://arxiv.org/abs/2505.19190) find
  38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  39. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. Preprint, arXiv:2308.02490.
  40. Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, Ludwig Schmidt, and Serena Yeung-Levy. 2025. Automated generation of challenging multiple-choice questions for vision language model evaluation. In Conference on C… find
  41. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. Preprint, arXiv:1511.03416.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-cogr-moe-concept-guided-expert/