Elamur Research⌕
☰
home / papers / 2026 / Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing

Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing

Paper · 2026 · Chunyu Ye, Yunhao Zhang, Jingyuan Sun et al.

Summary

Most brain-to-text decoders map neural recordings onto a single representation type, such as image or text embeddings. This ignores the fact that perception is cross-modal: reading about a barking dog can also evoke sights and sounds. Ye et al. build a unified decoder on Qwen2.5-Omni that addresses this gap.

The decoder has one projector per modality (text, image, audio). Each projector maps brain signals toward embeddings produced by auxiliary encoders, which are CLIP-ViT for images, Whisper-large-v3 for audio, and a query-based text encoder. These encoders are used only during training. A Modality Router then weights and fuses the projector outputs for each sample, and the LLM generates text directly from the fused brain features.

Training has two phases: multimodal instruction tuning, then projector fusion. The authors evaluate on three fMRI datasets (NSD, Pereira, Huth) and on MEG and EEG. The model reaches state-of-the-art results, including an 8.48% average gain on NSD.

This matters because it is presented as the first single BCI architecture that handles varied stimulus types and recording techniques.

Key points

  • Data: The fMRI benchmarks span three kinds of stimuli: images in NSD, written text in Pereira and spoken audio in Huth. The method is also tested on MEG and EEG, which trade spatial resolution for temporal resolution.
  • Unimodal baselines: On NSD the model beats approaches using a single brain representation by 8.48% on average. UMBRAE still wins on the CLIP-based metric but is weaker on text metrics.
  • Modality ablation: The fused "omni" variant beats every unimodal version, with average relative gains of 5.42% (NSD), 7.81% (Pereira) and 10.96% (Huth). Among single modalities, vision is strongest and audio weakest.
  • MEG and EEG: The method outperforms the mismatched, linear and BrainLLM baselines on all metrics. BLEU-1 reaches 20.96 on MEG and 20.03 on EEG, versus 19.59 and 18.29 for BrainLLM.
  • Router behaviour: Router weights favour the modality of the stimulus (watching, reading or listening). Textual weights rise for abstract items and visual weights for concrete ones. A load-balancing loss keeps the router from collapsing onto one modality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Chunyu Ye, Yunhao Zhang, Jingyuan Sun, Chong Li, Yang Zhao and Shaonan Wang. 2026. Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1131

@inproceedings{ye2026decoding,
  title = {{Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing}},
  author = {Chunyu Ye, Yunhao Zhang, Jingyuan Sun, Chong Li, Yang Zhao and Shaonan Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1131},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1131}
}

References

42 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716– 23736.
  2. Ethical Statements This study used preprocessed data from publicly available datasets. The data were de-identified/anonymized as part of the original dataset releases and did not include direct personal identifiers. The original collection of the fMRI, MEG, and EEG data was conducted under appropria… find
  3. Andrew J Anderson, Douwe Kiela, Stephen Clark, and Massimo Poesio. 2017. Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns. Transactions of the Association for Computational Linguistics, 5:17–30. find
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  5. Lawrence W Barsalou. 2008. Grounded cognition. Annu. Rev. Psychol., 59(1):617–645. find
  6. Joshua Bolam, Stephanie C Boyle, Robin AA Ince, and Ioannis Delis. 2022. Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions. NeuroImage, 247:118841. find
  7. Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin. 2023. Dewave: Discrete eeg waves encoding for brain dynamics to text translation. arXiv preprint arXiv:2309.14030. — [arXiv:2309.14030](https://arxiv.org/abs/2309.14030) find
  8. Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. 2023. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560. — [arXiv:2305.11560](https://arxiv.org/abs/2305.11560) find
  9. Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584–26595. find
  10. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718.
  11. Wanqiu Huang, Ke Ma, Tingyu Xie, and Hongwei Wang. 2025. Brainchat: Interactive semantic information decoding from fmri using large-scale vision-language pretrained models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  12. Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):963. find
  13. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
  14. Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, and Rex Ying. 2025. Mindllm: A subject-agnostic and versatile model for fmri-to-text decoding. arXiv preprint arXiv:2502.15786. — [arXiv:2502.15786](https://arxiv.org/abs/2502.15786) find
  15. Jennifer A Johnson and Robert J Zatorre. 2005. Attention to simultaneous unrelated auditory and visual events: behavioral and neural correlates. Cerebral cortex, 15(10):1609–1620. find
  16. Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. 2023. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555. find
  17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  19. Hanwen Liu, Daniel Hajialigol, Benny Antony, Aiguo Han, and Xuan Wang. 2024. Eeg2text: Open vocabulary eeg-to-text decoding with eeg pre-training and multi-view transformer. arXiv preprint arXiv:2405.02165. — [arXiv:2405.02165](https://arxiv.org/abs/2405.02165) find
  20. Weijian Mai and Zhijun Zhang. 2023. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428. — [arXiv:2308.07428](https://arxiv.org/abs/2308.07428) find
  21. Martin Meyer, Simon Baumann, Sarah Marchina, and Lutz Jancke. 2007. Hemodynamic responses in human multisensory and auditory association cortex to purely visual stimulation. BMC neuroscience, 8(1):14. find
  22. Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. science, 320(5880):1191–1195. find
  23. Allan Paivio. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue canadienne de psychologie, 45(3):255. find
  24. Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. 2009. Zero-shot learning with semantic output codes. Advances in neural information processing systems, 22. find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  26. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.
  27. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  28. Mor Regev, Christopher J Honey, Erez Simony, and Uri Hasson. 2013. Selective and invariant neural responses to spoken and written narratives. Journal of Neuroscience, 33(40):15978–15988. find
  29. Charles Spence. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73(4):971–995. find
  30. Yu Takagi and Shinji Nishimoto. 2023. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14453–14463. find
  31. Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. 2023. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858–866. find
  32. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
  33. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  34. Shaonan Wang, Xiaohan Zhang, Jiajun Zhang, and Chengqing Zong. 2022. A synchronized multimodal neuroimaging dataset for studying brain language processing. Scientific Data, 9(1):590. find
  35. Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. 2024. Umbrae: Unified multimodal brain decoding. In European Conference on Computer Vision, pages 242–259. Springer. find
  36. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  37. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024a. Qwen2 technical report. Preprint, arXiv:24… find
  38. Yiqian Yang, Hyejeong Jo, Yiqun Duan, Qiang Zhang, Jinni Zhou, Won Hee Lee, Renjing Xu, and Hui Xiong. 2024b. Mad: Multi-alignment meg-to-text decoding. arXiv preprint arXiv:2406.01512. — [arXiv:2406.01512](https://arxiv.org/abs/2406.01512) find
  39. Ziyi Ye, Qingyao Ai, Yiqun Liu, Maarten de Rijke, Min Zhang, Christina Lioma, and Tuukka Ruotsalo. 2025. Generative language reconstruction from brain recordings. Communications Biology, 8(1):346. find
  40. Xinpei Zhao, Jingyuan Sun, Shaonan Wang, Jing Ye, Xiaohan Zhang, and Chengqing Zong. 2024. Mapguide: A simple yet effective method to reconstruct continuous language from brain activities. arXiv preprint arXiv:2403.17516. find
  41. Yuxiao Zhao, Guohua Dong, Lei Zhu, and Xiaomin Ying. 2025. Memory recall: Retrieval-augmented mind reconstruction for brain decoding. Information Fusion, page 103280. find
  42. Shuxian Zou, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2022. Cross-modal cloze task: A new task to brain-to-word decoding. In Findings of the Association for Computational Linguistics: ACL 2022, pages 648–657. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ye-2026-decoding-multimodal-mind-generalizable-brain/