Elamur Research⌕
☰
home / papers / 2026 / Language Reconstruction with Brain Predictive Coding from fMRI Data

Language Reconstruction with Brain Predictive Coding from fMRI Data

Paper · 2026 · Congchi Yin, Ziyi Ye and Piji Li

Summary

This work asks whether the brain's habit of anticipating upcoming words can improve the decoding of continuous language from fMRI recordings. The authors first ran a verification study. They used ridge regression to map GPT-2 activations onto brain responses and found that adding representations of future words raises this correlation. The gain depends on how far ahead the predicted words are and how many are included, and it is strongest in prediction-related regions of interest (ROIs). Building on this, they propose PredFT, an end-to-end fMRI-to-text model with two parts:

  • Main network: an fMRI encoder (a 3D-CNN for volumetric data or linear layers for surface data), an FIR model to correct for BOLD delay, and a Transformer encoder-decoder that generates text.
  • Side network: encodes signals from predictive ROIs and is trained to output future words. Its representation is fed into the main decoder through attention.

On the LeBel and Narratives datasets, PredFT beats existing decoders on several language metrics. The results suggest that neuroscience theory can usefully guide brain-to-language model design.

Key points

  • The verification experiment defines a prediction score: the increase in brain score when embeddings of future words, at a given distance d and length l, are concatenated to the current word's representation. Medium lengths (around l = 4–6) gave the highest scores.
  • PredFT is trained jointly with two cross-entropy losses, one on the original word sequence (main network) and one on the predicted future words (side network).
  • Feeding the side network signals from the parietal-temporal-occipital (PTO) area produced larger decoding gains than other ROI choices, supporting this area's role in prediction.
  • On LeBel's dataset (3 subjects), the authors swept l from 1–12 and d from 0–12. On Narratives, they fixed l = 2 and varied d from 0–10 because of compute cost. Decoding was evaluated with BLEU-1 and ROUGE1-F.
  • Decoding quality first rises and then falls as d grows. Very short (l = 1) or very long (l = 11–12) predictions hurt performance. This is a stricter pattern than in the verification study, where such lengths still gave small gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Nanjing University of Aeronautics and AstronauticsNanjing · China4 works · 2 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

Cite

Cite as Congchi Yin, Ziyi Ye and Piji Li. 2026. Language Reconstruction with Brain Predictive Coding from fMRI Data. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.949

@inproceedings{yin2026language,
  title = {{Language Reconstruction with Brain Predictive Coding from fMRI Data}},
  author = {Congchi Yin, Ziyi Ye and Piji Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.949},
  url = {https://doi.org/10.18653/v1/2026.acl-long.949}
}

References

49 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics. find
  2. Nicolas Affolter, Beni Egressy, Damian Pascual, and Roger Wattenhofer. 2020. Brain2word: decoding brain activity for language generation. arXiv preprint arXiv:2009.04765. — [arXiv:2009.04765](https://arxiv.org/abs/2009.04765) find
  3. Richard Antonello and Alexander Huth. 2024. Predictive coding or just feature discovery? an alternative account of why language models fit brain data. Neurobiology of Language, 5(1):64–79. find
  4. Shohini Bhattasali, Murielle Fabre, Wen-Ming Luh, Hazem Al Saied, Mathieu Constant, Christophe Pallier, Jonathan R Brennan, R Nathan Spreng, and John Hale. 2019. Localising memory retrieval and syntactic composition: an fmri study of naturalistic language comprehension. Language, Cognition and Neuro… find
  5. Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2022. Deep language algorithms predict semantic comprehension from brain activity. Scientific Reports, 12(1):16327. find
  6. Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2023. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, 7(3):430–441. find
  7. Christophe Destrieux, Bruce Fischl, Anders Dale, and Eric Halgren. 2010. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage, 53(1):1–15. find
  8. Peter W Donhauser and Sylvain Baillet. 2020. Two distinct neural timescales for predictive speech processing. Neuron, 105(2):385–393. find
  9. Karl Friston. 2008. Hierarchical models in the brain. PLoS Computational Biology, 4(11):e1000211. find
  10. Karl Friston and Stefan Kiebel. 2009. Predictive coding under the free-energy principle. Philosophical transactions of the Royal Society B: Biological sciences, 364(1521):1211–1221. find
  11. Marta I Garrido, James M Kilner, Klaas E Stephan, and Karl J Friston. 2009. The mismatch negativity: a review of underlying mechanisms. Clinical Neurophysiology, 120(3):453–463. find
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  13. Micha Heilbron, Kristijan Armeni, Jan-Mathijs Schoffelen, Peter Hagoort, and Floris P De Lange. 2022. A hierarchy of linguistic predictions during natural language comprehension. Proceedings of the National Academy of Sciences, 119(32):e2201968119. find
  14. Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453– 458. find
  15. Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. 2023. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555. find
  16. Yann LeCun and Yoshua Bengio. Convolutional networks for images, speech, and time series. The Handbook of Brain Theory and Neural Networks. find
  17. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  18. Chien Heng Liao, Keith J Worsley, J-B Poline, John AD Aston, Gary H Duncan, and Alan C Evans. 2002. Estimating the delay of the fmri response. Neuroimage, 16(3):593–606. find
  19. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out.
  20. James L McClelland and David E Rumelhart. 1981. An interactive activation model of context effects in letter perception: I. an account of basic findings. Psychological Review, 88(5):375. find
  21. Beren Millidge, Anil Seth, and Christopher L Buckley. 2021. Predictive coding: a theoretical and experimental review. arXiv preprint arXiv:2107.12979. — [arXiv:2107.12979](https://arxiv.org/abs/2107.12979) find
  22. David Mumford. 1991. On the computational architecture of the neocortex: I. the role of the thalamo-cortical loop. Biological Cybernetics, 65(2):135– 145. find
  23. Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. 2021. The "narratives" fmri dataset for evaluating models of naturalistic language comprehension. Scientific Data, 8(1):250. find
  24. Kayoko Okada, William Matchin, and Gregory Hickok. 2018. Neural evidence for predictive coding in auditory cortex during speech production. Psychonomic Bulletin & Review, 25:423–430. find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.
  26. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research. find
  27. Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature Communications, 9(1):963. find
  28. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. find
  29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  30. Rajesh PN Rao and Dana H Ballard. 1999. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79–87. find
  31. Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. 2024. Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. In Advances in Neural Information Pro… find
  32. Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019. Towards sentence-level brain decoding with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems.
  34. Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. 2023. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858–866. find
  35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  36. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
  37. Catherine Wacongne, Etienne Labyt, Virginie Van Wassenhove, Tristan Bekinschtein, Lionel Naccache, and Stanislas Dehaene. 2011. Evidence for a hierarchy of predictions and prediction errors in human cortex. Proceedings of the National Academy of Sciences, 108(51):20754–20759. find
  38. Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. 2020. Fine-grained neural decoding with distributed word representations. Information Sciences, 507:256–272. find
  39. Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. 2024. Mindbridge: A cross-subject brain decoding framework. arXiv preprint arXiv:2404.07850. — [arXiv:2404.07850](https://arxiv.org/abs/2404.07850) find
  40. Roel M Willems, Stefan L Frank, Annabel D Nijhof, Peter Hagoort, and Antal Van den Bosch. 2016. Prediction during natural language comprehension. Cerebral Cortex, 26(6):2506–2516. find
  41. Yuxin Wu and Kaiming He. 2018. Group normalization. In Proceedings of the European Conference on Computer Vision. find
  42. Nuwa Xi, Sendong Zhao, Haochun Wang, Chi Liu, Bing Qin, and Ting Liu. 2023. Unicorn: Unified cognitive signal reconstruction bridging cognitive signals and human language. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.
  43. Runze Xia, Congchi Yin, and Piji Li. 2024. Decoding the echoes of vision from fmri: Memory disentangling for past semantic information. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 2040–2052. find
  44. Ziyi Ye, Qingyao Ai, Yiqun Liu, Min Zhang, Christina Lioma, and Tuukka Ruotsalo. 2023. Language generation from human brain activities. arXiv preprint arXiv:2311.09889. — [arXiv:2311.09889](https://arxiv.org/abs/2311.09889) find
  45. Congchi Yin, Qian Yu, Zhiwei Fang, Jie He, Changping Peng, Zhangang Lin, Jingping Shao, and Piji Li. 2023. Data contamination issues in brain-to-text decoding. arXiv preprint arXiv:2312.10987. — [arXiv:2312.10987](https://arxiv.org/abs/2312.10987) find
  46. Congchi Yin, Yongpeng Zhang, Xuyun Wen, and Piji Li. 2025. Improve language model and brain alignment via associative memory. In Findings of the Association for Computational Linguistics: ACL 2025, pages 986–999. find
  47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  48. Xinpei Zhao, Jingyuan Sun, Shaonan Wang, Jing Ye, Xiaohan Zhang, and Chengqing Zong. 2024. MapGuide: A simple yet effective method to reconstruct continuous language from brain activities. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui… find
  49. Shuxian Zou, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2021. Towards brain-to-text generation: Neural decoding with pre-trained encoder-decoder models. In Proceedings of the NeurIPS Workshop on AI for Science. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yin-2026-language-reconstruction-brain-predictive-coding/