Elamur Research⌕
☰
home / papers / 2026 / Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Paper · 2026 · Haolei Xu, Haiwen Hong, Hongxing Li et al.

Summary

The paper examines a failure mode in multimodal Mixture-of-Experts (MoE) vision-language models. The authors call it "Seeing but Not Thinking": a model reads an image-rendered problem correctly but then reasons wrongly, even though it solves the same problem when given as text. Using image-rendered versions of MATH500, they confirm that most image-only failures are reasoning failures, not misreadings. A hidden-state concept-swapping experiment shows that visual and textual meanings do align in the middle layers of MoE models, so poor alignment alone does not explain the gap. Routing analysis points to a different cause. Vision-oriented experts sit mainly in early and final layers, while domain reasoning experts sit in middle layers. Image inputs shift routing away from text-like patterns exactly in those middle layers. The authors call this the Routing Distraction hypothesis. They test it with an inference-time intervention that boosts the routing weights of domain experts. These experts are found by checking which ones activate more often on domain-specific text than on general text. The soft variant improves accuracy across three models and six benchmarks. This suggests a cheap way to recover reasoning ability in multimodal MoE models without retraining.

Key points

  • Error analysis: On cases solved as text but failed as images, 68.2–73.1% of errors come from reasoning and only 26.9–31.8% from perception.
  • Concept intervention: Success rates follow an inverted U-shape across layers. In Qwen3-VL-30B-A3B, layers 8–42 exceed 90%, which indicates cross-modal semantic sharing in MoE.
  • Models and benchmarks: The method was tested on Kimi-VL-16B-A3B, Qwen3-VL-30B-A3B and Llama4-Scout-109B-A17B. The benchmarks were image-rendered MATH500, GPQA-Diamond (chemistry and physics subsets), MathVerse, MATH-Vision and GSM8K-V.
  • Results: Soft routing guidance raises average accuracy by +2.36 (Kimi-VL), +1.48 (Qwen3-VL) and +1.17 (Llama4) points. The largest single gain is +3.17 on MathVerse with Kimi-VL. Random and hard guidance mostly reduce accuracy.
  • Transfer and tuning: Experts identified from GSM8K text still help on MathVerse, which suggests they capture reasoning functions rather than sample-specific answers. The best enhancement strength λ is about 0.4–0.6, but Llama4 needs about 0.2 because of its Top-1 routing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu and Yueting Zhuang. 2026. Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1438

@inproceedings{xu2026seeing,
  title = {{Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts}},
  author = {Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu and Yueting Zhuang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1438},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1438}
}

References

45 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  2. Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz, Junlin Hu, and Nanyun Peng. 2025. Multilingual routing in mixture-of-experts. arXiv preprint arXiv:2510.04694. — [arXiv:2510.04694](https://arxiv.org/abs/2510.04694) find
  3. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering. find
  4. Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. 2025. xverify: Efficient answer verifier for reasoning model evaluations. arXiv preprint arXiv:2504.10481. — [arXiv:2504.10481](https://arxiv.org/abs/2504.10481) find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  6. Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, and 1 others. 2026. Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models. arXiv preprint arXiv:2602.04804. — [arXiv:2602.04804](https://arxiv.org/abs/2602.04804) find
  7. Yue Ding, Xiaofang Zhu, Tianze Xia, Junfei Wu, Xinlong Chen, Qiang Liu, and Liang Wang. 2025. D 2 hscore: Reasoning-aware hallucination detection via semantic breadth and depth analysis in llms. arXiv preprint arXiv:2509.11569. — [arXiv:2509.11569](https://arxiv.org/abs/2509.11569) find
  8. Yanrui Du, Sendong Zhao, Yibo Gao, Danyang Zhao, Qika Lin, Ming Ma, Jiayun Li, Yi Jiang, Kai He, Qianyi Xu, and 1 others. 2026. S3-cot: Self-sampled succinct reasoning enables efficient chain-of-thought llms. arXiv preprint arXiv:2602.01982. find
  9. Mohsen Fayyaz, Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Ryan Rossi, Trung Bui, Hinrich Schütze, and Nanyun Peng. 2025. Steering moe llms via expert (de) activation. arXiv preprint arXiv:2509.09660. — [arXiv:2509.09660](https://arxiv.org/abs/2509.09660) find
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  11. Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403.07652. — [arXiv:2403.07652](https://arxiv.org/abs/2403.07652) find
  12. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987. find
  13. Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. 2025. Natural language understanding and inference with mllm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36. find
  14. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  15. Zhenglin Lai, Mengyao Liao, Bingzhe Wu, Dong Xu, Zebin Zhao, Zhihang Yuan, Chao Fan, and Jianqiang Li. 2025. Safex: Analyzing vulnerabilities of moe-based llms via stable safety-critical expert identification. arXiv preprint arXiv:2506.17368. — [arXiv:2506.17368](https://arxiv.org/abs/2506.17368) find
  16. Zhongyang Li, Ziyue Li, and Tianyi Zhou. 2025. R2-t2: Re-routing in test-time for multimodal mixture-of-experts. arXiv preprint arXiv:2502.20395. — [arXiv:2502.20395](https://arxiv.org/abs/2502.20395) find
  17. Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612–17625. find
  18. Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, and 1 others. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947. — [arXiv:2401.15947](https://arxiv.org/abs/2401.15947) find
  19. Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2025. A closer look into mixture-of-experts in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4427–4447. find
  20. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Pengxiang Zhao, Guangyi Liu, and 1 others. 2026a. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40,… find
  21. Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. 2026b. Skill0: In-context agentic reinforcement learning for skill internalization. arXiv preprint arXiv:2604.02268. — [arXiv:2604.02268](https://arxiv.org/abs/2604.02268) find
  22. Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, and 1 others. 2025. Ui-s1: Advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11543. — [arXiv:2509.11543](https://arxiv.org/abs/2509.11543) find
  23. Grace Luo, Trevor Darrell, and Amir Bar. 2024. Vision-language models create cross-modal task representations. arXiv preprint arXiv:2410.22330. find
  24. Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2022. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162. find
  25. Meta AI. 2025. Introducing llama 4: Advancing multimodal intelligence. Meta AI Blog. find
  26. Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models. In Proceedings of the 63rd Annual Meeting of the Association f…
  27. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  28. Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. 2024. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. arXiv preprint arXiv:2404.07983. — [arXiv:2404.07983](https://arxiv.org/abs/2404.07983) find
  29. Chufan Shi, Cheng Yang, Xinyu Zhu, Jiahao Wang, Taiqiang Wu, Siheng Li, Deng Cai, Yujiu Yang, and Yu Meng. 2024. Unchosen experts can contribute too: Unleashing moe models’ power by self-contrast. Advances in Neural Information Processing Systems, 37:136897–136921. find
  30. Mustafa Shukor and Matthieu Cord. 2024. Implicit multimodal alignment: On the generalization of frozen llms to multimodal inputs. Advances in Neural Information Processing Systems, 37:130848–130886. find
  31. Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, and 1 others. 2025. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865. — [arXiv:2504.13865](https://arxiv.org/abs/2504.13865) find
  32. Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, and 1 others. 2025. Kimi-vl technical report. arXiv preprint arXiv:2504.07491.
  33. Angela van Sprang, Laurens Samson, Ana Lucic, Erman Acar, Sennay Ghebreab, and Yuki M Asano. 2025. Same content, different answers: Cross-modal inconsistency in mllms. arXiv preprint arXiv:2512.08923. — [arXiv:2512.08923](https://arxiv.org/abs/2512.08923) find
  34. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169.
  35. Mengru Wang, Xingyu Chen, Yue Wang, Zhiwei He, Jiahao Xu, Tian Liang, Qiuzhi Liu, Yunzhi Yao, Wenxuan Wang, Ruotian Ma, and 1 others. 2025a. Two experts are all you need for steering thinking: Reinforcing cognitive effort in moe reasoning models without additional training. arXiv preprint arXiv:2505… — [arXiv:2505.14681](https://arxiv.org/abs/2505.14681) find
  36. Kai-Wei Chang, Yu Qiao, and 1 others. 2024a. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer.
  37. Xiang Zhang, Senyu Li, Ning Shi, Bradley Hauer, Zijun Wu, Grzegorz Kondrak, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. 2024b. Cross-modal consistency in multimodal large language models. arXiv preprint arXiv:2411.09273. — [arXiv:2411.09273](https://arxiv.org/abs/2411.09273) find
  38. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025b. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  39. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394.
  40. Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2025. SHARP: Steering hallucination in LVLMs via representation engineering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14346–143… find
  41. Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. 2024. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. arXiv preprint arXiv:2411.04986. find
  42. Haolei Xu, Xinyu Mei, Yuchen Yan, Rui Zhou, Wenqi Zhang, Weiming Lu, Yueting Zhuang, and Yongliang Shen. 2025. Easysteer: A unified framework for high-performance and extensible llm steering. arXiv preprint arXiv:2509.25175. — [arXiv:2509.25175](https://arxiv.org/abs/2509.25175) find
  43. Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739.
  44. Fan Yuan, Yuchen Yan, Yifan Jiang, Haoran Zhao, Tao Feng, Jinyan Chen, Yanwei Lou, Wenqi Zhang, Yongliang Shen, Weiming Lu, and 1 others. 2025. Gsm8k-v: Can vision language models solve grade school math word problems in visual contexts. arXiv preprint arXiv:2509.25160. — [arXiv:2509.25160](https://arxiv.org/abs/2509.25160) find
  45. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-seeing-but-not-thinking-routing/