Elamur Research⌕
☰
home / papers / 2026 / Topic-context Dependency on Continuous Semantic Reconstruction of Language from fMRI Signals

Topic-context Dependency on Continuous Semantic Reconstruction of Language from fMRI Signals

Summary

Brain-to-text decoders combine an LLM that proposes candidate continuations with a voxelwise encoder that predicts fMRI responses to each candidate. This work asks whether the topical makeup of the encoder's training stories skews what gets decoded. The authors used the LeBel et al. (2023) natural-listening dataset: 84 podcast stories heard by three participants. They embedded the transcripts with text-embedding-3-small and split them with K-Means into a Family cluster and a Miscellaneous cluster. They then trained GPT-2-based encoders on one topic only and reconstructed stories following the Tang et al. (2023) decoding approach. Outputs were judged in two ways: by re-clustering the decoded text, and by BERTScore against the true transcripts.

The encoders pulled reconstructions toward their own training topic. Family-trained encoders kept Family stories in the Family cluster, while Miscellaneous-trained encoders moved them out of it. Decoding beat chance only within the training topic and dropped below a random baseline across topics. The authors conclude that encoders can over-specialize semantically, which matters for brain-computer interfaces and for how decoding results are interpreted. They argue for training stimuli that are more diverse and lexically richer.

Key points

  • Data: the LeBel et al. (2023) fMRI corpus of 77 The Moth Radio Hour and 7 Modern Love stories. Participants S1–S3 each heard all stories, more than 16 hours of audio.
  • Clustering: 1536-dimensional OpenAI text-embedding-3-small vectors were split by K-Means (K=2, silhouette 0.37). Eight Miscellaneous stories were dropped to give 38 stories per cluster.
  • Encoder: GPT-2 Small with word-level tokenization, using layer 9 features and a five-word context. Leave-one-out training (37 stories per fold) produced 114 per-fold models plus 6 whole-cluster models. Compute totaled roughly 2,000 hours.
  • Topic bias: with Family-trained encoders, 36 of 38 Family stories stayed in the Family cluster for at least one participant. With Miscellaneous-trained encoders, every reconstruction moved to the Miscellaneous cluster.
  • Fidelity: BERTScore was computed with microsoft/deberta-xlarge-mnli. Topic-constrained encoders beat a random encoder only on in-topic stories and fell below it on out-of-topic ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/233sezzfxz2m

@inproceedings{travi2026topic,
  title = {{Topic-context Dependency on Continuous Semantic Reconstruction of Language from fMRI Signals}},
  author = {Travi, Fermin and Delmagro, Agustin and Slezak, Diego Fernandez and Bianchi, Bruno and Kamienkowski, Juan E.},
  booktitle = {Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics},
  year = {2026},
  doi = {10.63317/233sezzfxz2m},
  url = {https://doi.org/10.63317/233sezzfxz2m}
}

References

21 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Richard Antonello and Alexander Huth. 2024. Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data. Neurobiology of Language (Cambridge, Mass.), 5(1):64–79. find
  2. Richard Antonello, Javier Turek, Vy Vo, and Alexander Huth. 2021. Low-dimensional structure in the space of language representations is reflected in brain responses. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, pages 8332–8344, Red Hook, NY,… find
  3. Richard Antonello, Aditya Vaidya, and Alexander Huth. 2023. Scaling laws for language encoding models in fMRI. Advances in Neural Information Processing Systems, 36:21895–21907. find
  4. Catherine Chen, Tom Dupré la Tour, Jack L. Gallant, Daniel Klein, and Fatma Deniz. 2024. The cortical representation of language timescales is shared between reading and listening. Communications Biology, 7(1):284. Publisher: Nature Publishing Group UK London. find
  5. Jiaqi Chen, Richard Antonello, Kaavya Chaparala, Coen Arrow, and Nima Mesgarani. 2025. Quantifying Semantic Functional Specialization in the Brain Using Encoding Models of Natural Language. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 77–90, Albuquerque,… find
  6. Srikanth R. Damera, Lillian Chang, Plamen P. Nikolov, James A. Mattei, Suneel Banerjee, Laurie S. Glezer, Patrick H. Cox, Xiong Jiang, Josef P. Rauschecker, and Maximilian Riesenhu-ber. 2023. Evidence for a Spoken Word Lexicon in the Auditory Ventral Stream. Neurobiology of Language, 4(3):420–434. find
  7. Tom Dupré la Tour, Matteo Visconti di Oleg-gio Castello, and Jack L. Gallant. 2025. The Voxelwise Encoding Model framework: A tutorial introduction to fitting encoding models to fMRI data. Imaging Neuroscience, 3:imag_a_00575. find
  8. Evelina Fedorenko, Anna A. Ivanova, and Tamar I. Regev. 2024. The language network as a natural kind within the broader landscape of the human brain. Nature Reviews Neuroscience, 25(5):289– 312. find
  9. Laura Gwilliams, Alec Marantz, David Poeppel, and Jean-Rémi King. 2025. Hierarchical dynamic coding coordinates speech comprehension in the human brain. Proceedings of the National Academy of Sciences, 122(42):e2422097122. find
  10. Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G. Huth. 2023. A natural language fMRI dataset for voxelwise encoding models. Scientific Data, 10(1):555. Publisher: Nature Publishing Group. find
  11. Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, Neil D. Lawrence, Michael I. Jordan, and Thomas Dietterich, editors. 2022. find
  12. Dataset Shift in Machine Learning. Neural Information Processing series. MIT Press, Cambridge, MA, USA. find
  13. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. find
  14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. find
  15. Alexander Schaefer, Ru Kong, Evan M. Gordon, Timothy O. Laumann, Xi-Nian Zuo, Avram J. Holmes, Simon B. Eickhoff, and B. T. Thomas Yeo. 2018. Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI. find
  16. Cerebral Cortex (New York, N.Y.: 1991), 28(9):3095–3114. find
  17. Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth. 2023. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858–866. Publisher: Nature Publishing Group. find
  18. Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  19. Aditya Vaidya, Javier Turek, and Alexander Huth. 2023. Humans and language models diverge when predicting repeating text. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 58–69, Singapore. Association for Computational Linguistics. find
  20. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. ArXiv:1904.09675 [cs]. find
  21. Yuchen Zhou, Emmy Liu, Graham Neubig, Michael J. Tarr, and Leila Wehbe. 2024. Divergences between Language Models and Human Brains. Advances in Neural Information Processing Systems, 37:137999–138031. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/travi-2026-topic-context-dependency-continuous-semantic/