Elamur Research⌕
☰
home / papers / 2026 / What Brain Data Adds to Language Model Training

What Brain Data Adds to Language Model Training

Summary

Earlier studies reported that fine-tuning language models to predict fMRI responses ("brain-tuning") improves some downstream tasks. However, those baselines usually also saw less of the stimulus text, so the gains could come from extra exposure to that text. The authors set up a controlled comparison of three regimes on bert-base-cased and gpt2-small:

  • Brain-Tuned: trained only to predict brain recordings through a brain mapping head.
  • Stimulus-Tuned: trained only on the matching stimulus text.
  • Jointly-Tuned: trained on both signals together.

Training uses two public fMRI reading datasets: Harry Potter and The Moth Radio Hour. Each model is then probed for linguistic competence with the Holmes benchmark, which has over 200 tasks covering syntax, semantics, discourse, morphology and reasoning.

Joint tuning beats the pretrained models in every subfield, and Brain-Tuned models beat their Stimulus-Tuned counterparts. Single-signal tuning, however, usually falls below the pretrained baseline. The study shows that neural recordings carry useful signal beyond the text itself and offers a general way to measure such non-textual information gain.

Key points

  • Datasets:
    • Wehbe et al. (2014a): 8 participants reading Chapter 9 of Harry Potter and the Sorcerer's Stone, word by word at 500 ms each, TR=2s, 1211 images per participant.
    • Deniz et al. (2019): 6 participants, 11 Moth Radio Hour stories, 4028 images, reading condition used.
  • Evaluation: Holmes (Waldis et al., 2024) with more than 200 probing datasets grouped by linguistic subfield.
  • Jointly-Tuned results: these were the only models that consistently surpassed the pretrained baselines, with the clearest gains in syntax, morphology and discourse.
  • Brain-Tuned vs. Stimulus-Tuned: Brain-Tuned models outperformed Stimulus-Tuned ones. Both were generally worse than pretrained models, and Brain-Tuned exceeded pretrained only on morphology.
  • Code: github.com/bridge-ai-neuro/lm-brain-tuning. Stated limitations are incomplete benchmark coverage (e.g., few discourse datasets), only two fMRI datasets, and a single stimulus genre.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.12

@inproceedings{merlin2026what,
  title = {{What Brain Data Adds to Language Model Training}},
  author = {Merlin, Gabriele and Moussa, Omer and Toneva, Mariya},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.12},
  url = {https://doi.org/10.18653/v1/2026.conll-main.12}
}

References

44 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mostafa Abdou, Ana Valeria González, Mariya Toneva, Daniel Hershcovich, and Anders Søgaard. 2021. Does injecting linguistic structure into language models lead to better alignment with brain recordings? arXiv preprint arXiv:2101.12608. — [arXiv:2101.12608](https://arxiv.org/abs/2101.12608) find
  2. Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. 2025. From language to cognition: How llms outgrow the human language network. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  3. Samuel Amouyal, Aya Meltzer-Asscher, and Jonathan Berant. 2024. Large language models for psycholinguistic plausibility pretesting. In Findings of the Association for Computational Linguistics: EACL 2024, pages 166–181, St. Julian’s, Malta. Association for Computational Linguistics. find
  4. Khai Loong Aw and Mariya Toneva. 2023. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations. find
  5. Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2767–2796. find
  6. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2023. Prompting language models for linguistic structure. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6649–6663, Toronto, Canada. Association for Computational Linguistics. find
  7. Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. 2021. Disentangling syntax and semantics in the brain with deep networks. In International conference on machine learning, pages 1336–1348. PMLR. find
  8. Fatma Deniz, Anwar O Nunez-Elizalde, Alexander G Huth, and Jack L Gallant. 2019. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39):7722– 7736. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,… find
  10. E. Fedorenko, P.-J. Hsieh, A. Nieto-Castanon, S. Whitfield-Gabrieli, and N. Kanwisher. 2010. New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. Journal of Neurophysiology, 104(2):1177–1194. find
  11. Evelina Fedorenko and Sharon L Thompson-Schill. 2014. Reworking the language network. Trends in cognitive sciences, 18(3):120–126. find
  12. Maëlle Freteault, Maximilien Le Clei, Loic Tetrel, Lune Bellec, and Nicolas Farrugia. 2025. Alignment of auditory artificial networks with massive individual fmri brain data leads to generalisable improvements in brain encoding and downstream tasks. Imaging Neuroscience, 3. find
  13. Jon Gauthier and Roger Levy. 2019. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 529–539, Hong… find
  14. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, and 1 others. 2022. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):369–380. find
  15. Anuja Negi, Subba Reddy Oota, Anwar O Nunez-Elizalde, Manish Gupta, and Fatma Deniz. 2025. Brain-informed fine-tuning for improved multilingual understanding in language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  16. Eghbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. 2024. Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language, 5(1):43–63. find
  17. Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. 2011. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):1641–1646. find
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  19. Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. 2024a. Speech language models lack important brain-relevant semantics. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8503–8528, Bangkok, Thailand. Association for… find
  20. Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453– 458. find
  21. Shailee Jain and Alexander Huth. 2018. Incorporating context into language encoding models for fMRI. In Advances in Neural Information Processing Systems, pages 6628–6637. find
  22. Subba Reddy Oota, Manish Gupta, and Mariya Toneva. 2024b. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36. find
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  24. Carina Kauf, Greta Tuckute, Roger Levy, Jacob Andreas, and Evelina Fedorenko. 2023. Lexical semantic content, not syntactic structure, is the main contributor to ann-brain similarity of fmri responses in the language network. bioRxiv. find
  25. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. find
  26. Aniketh Janardhan Reddy and Leila Wehbe. 2021. Can fmri reveal the representation of syntactic structure in the brain? Advances in Neural Information Processing Systems, 34:9843–9856. find
  27. J.K. Rowling, M. GrandPre, M. GrandPré, T. Taylor, Arthur A. Levine Books, and Scholastic Inc. 1998. Harry Potter and the Sorcerer’s Stone. Harry Potter. A.A. Levine Books. find
  28. Gabriele Merlin and Mariya Toneva. 2024. Language models and brains align due to more than next-word prediction and word-level information. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18431–18454, Miami, Florida, USA. Association for Computational… find
  29. Gabriele Merlin and Mariya Toneva. 2026. When language models lose their mind: The consequences of brain misalignment. In The Fourteenth International Conference on Learning Representations. find
  30. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45). find
  31. Dan Schwartz, Mariya Toneva, and Leila Wehbe. 2019. Inducing brain-relevant bias in natural language processing models. Advances in neural information processing systems, 32. find
  32. Omer Moussa, Dietrich Klakow, and Mariya Toneva. 2025. Improving semantic understanding in speech language models via brain-tuning. ICLR. find
  33. Mariya Toneva, Tom M Mitchell, and Leila Wehbe. 2022a. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745–757. find
  34. Omer Moussa and Mariya Toneva. 2025. Brain-tuning improves generalizability and efficiency of brain alignment in speech models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  35. Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in Neural Information Processing Systems, 32. find
  36. Mariya Toneva, Jennifer Williams, Anand Bollu, Christoph Dann, and Leila Wehbe. 2022b. Same cause; different effects in the brain. Causal Learning and Reasoning. find
  37. Nishitha Vattikonda, Aditya R. Vaidya, Richard J. Antonello, and Alexander G. Huth. 2025. Brainwavlm: Fine-tuning speech representations with brain responses to language. Preprint, arXiv:2502.08866. — [arXiv:2502.08866](https://arxiv.org/abs/2502.08866) find
  38. Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. 2024. Holmes a benchmark to assess the linguistic competence of language models. Transactions of the Association for Computational Linguistics, 12:1616–1647. find
  39. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32. find our card
  40. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  41. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. find
  42. Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. 2014a. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PloS one, 9(11):e112575. find
  43. Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. 2014b. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 233– 243. find
  44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf… find our card

↑ back to top

This page in Markdown →