Elamur Research⌕
☰
home / papers / 2026 / Temporal Precision Matters: Brain-Tuning Speech Language Models with Millisecond-Resolution Neural Signals

Temporal Precision Matters: Brain-Tuning Speech Language Models with Millisecond-Resolution Neural Signals

Paper · 2026 · Zhejun Zhang, Wenqing Zhou, Haozhe Xu et al.

Summary

Brain-tuning fine-tunes speech language models so that they predict human neural activity. Earlier work has relied almost entirely on fMRI. Because each fMRI sample integrates activity over about two seconds, several stages of language processing merge into one target. The authors propose ECoG-tuning, which instead supervises models with intracranial electrocorticography recorded at millisecond resolution.

They use the Podcast ECoG dataset and train on word-level high-gamma responses. Two supervision windows are anchored to word onset, based on known latency differences in neural encoding:

  • an early speech window for acoustic-phonetic processing;
  • a later language window for higher-level linguistic processing.

Hidden states from Wav2Vec 2.0, HuBERT and Whisper are pooled across layers. A linear map then predicts the full electrode-by-time response under an MSE loss.

ECoG-tuning raised brain alignment beyond both pretrained and distillation baselines. Keeping full temporal dynamics outperformed time-averaged targets, and language-window tuning helped most in language-responsive cortex. Downstream speech task performance stayed the same or improved. The study suggests electrophysiology is a workable brain-tuning signal and that neuroscience findings about timing can guide training design.

Key points

  • Data: Released by Zada et al. (2025), the Podcast ECoG dataset records 9 participants as they listen to a 30-minute podcast (5,137 timestamped words). After quality control, 1,268 electrodes remained. Targets are high-gamma power (70–200 Hz), giving about 5,000 word-level samples per participant.
  • Windows: The speech window spans −50 to +150 ms around word onset, and the language window spans +150 to +350 ms. Their boundary falls between the reported encoding peaks of about 54 ms (speech) and about 247 ms (language).
  • Models: Wav2Vec 2.0 Base, HuBERT Base and the Whisper-small encoder, each roughly 95–102M fine-tuned parameters with 12 layers. Input is 30 s of audio context, and the last 10 frames of every layer are pooled.
  • Main results: Full spatiotemporal supervision gave 7–17% higher alignment than time-averaged supervision. Downstream performance was kept or improved, with consistent gains on phoneme prediction.
  • Code: github.com/Mochizuki-BUPT/ECoG-Tuning-main.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beijing University of Posts and TelecommunicationsBeijing · China78 works · 33,365 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Zhejun Zhang, Wenqing Zhou, Haozhe Xu, Lin Zhang and Lei Li. 2026. Temporal Precision Matters: Brain-Tuning Speech Language Models with Millisecond-Resolution Neural Signals. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1911

@inproceedings{zhang2026temporal,
  title = {{Temporal Precision Matters: Brain-Tuning Speech Language Models with Millisecond-Resolution Neural Signals}},
  author = {Zhejun Zhang, Wenqing Zhou, Haozhe Xu, Lin Zhang and Lei Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1911},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1911}
}

References

47 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. 2025a. The LLM language network: A neuroscientific approach for identifying causally task-relevant units. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisti… find
  2. Badr AlKhamissi, Greta Tuckute, Yingtian Tang, Taha Osama A Binhuraib, Antoine Bosselut, and Martin Schrimpf. 2025b. From language to cognition: How LLMs outgrow the human language network. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 24332–24350,…
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates, Inc.
  4. Arnab Bhattacharjee, Zaid Zada, Haocheng Wang, Bobbi Aubrey, Werner Doyle, Patricia Dugan, Daniel Friedman, Orrin Devinsky, Adeen Flinker, Peter J. Ramadge, Uri Hasson, Ariel Goldstein, and Samuel A. Nastase. 2026. Aligning brains into a shared space improves their alignment with large language mode… find
  5. Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. 2014. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4):377–390. find
  6. Charlotte Caucheteux and Jean-Rémi King. 2022. Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1):134. find
  7. Nathan E. Crone, Alon Sinai, and Anna Korzeniewska. 2006. High-frequency gamma oscillations and human brain mapping with electrocorticography. In Progress in Brain Research, volume 159, pages 275– 295. find
  8. Xiao Ding, Bowen Chen, Li Du, Bing Qin, and Ting Liu. 2022. CogBERT: Cognition-guided pre-trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3210–3225, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  9. Berta Franzluebbers, Donald Dunagan, Miloš Stanojević, Jan Buys, and John Hale. 2024. Multipath parsing in the brain. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12215– 12229, Bangkok, Thailand. Association for Computatio… find
  10. Changjiang Gao, Jixing Li, Jiajun Chen, and Shujian Huang. 2024. Measuring meaning composition in the human brain with composition scores from large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11295– 11308… find
  11. John S. Garofolo, Lori F. Lamel, William M. Fisher, David S. Pallett, Nancy L. Dahlgren, Victor Zue, and Jonathan G. Fiscus. 1993. Timit acoustic-phonetic continuous speech corpus. find
  12. Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smith, and David C. Van Essen. 2016. A multimodal parcellation of human cerebral cortex. Nature, 536(7615):171–178. find
  13. Ariel Goldstein, Avigail Grinstein-Dabush, Mariano Schain, Haocheng Wang, Zhuoqiao Hong, Bobbi Aubrey, Samuel A. Nastase, Zaid Zada, Eric Ham, Amir Feder, Harshvardhan Gazula, Eliav Buchnik, Werner Doyle, Sasha Devore, Patricia Dugan, Roi Reichart, Daniel Friedman, Michael Brenner, Avinatan Hassidim… find
  14. Ariel Goldstein, Eric Ham, Mariano Schain, Samuel A. Nastase, Bobbi Aubrey, Zaid Zada, Avigail Grinstein-Dabush, Harshvardhan Gazula, Amir Feder, Werner Doyle, Sasha Devore, Patricia Dugan, Daniel Friedman, Michael Brenner, Avinatan Hassidim, Yossi Matias, Orrin Devinsky, Noam Siegelman, Adeen Flink… find
  15. Ariel Goldstein, Haocheng Wang, Leonard Niekerken, Mariano Schain, Zaid Zada, Bobbi Aubrey, Tom Sheffer, Samuel A. Nastase, Harshvardhan Gazula, Aditi Singh, Aditi Rao, Gina Choe, Catherine Kim, Werner Doyle, Daniel Friedman, Sasha Devore, Patricia Dugan, Avinatan Hassidim, Michael Brenner, and 4 ot… find
  16. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, and 13 others. 2022. Shared co… find
  17. Laura Gwilliams, Alec Marantz, David Poeppel, and Jean-Rémi King. 2025. Hierarchical dynamic coding coordinates speech comprehension in the human brain. Proceedings of the National Academy of Sciences, 122(42):e2422097122. find
  18. Linyang He, Ercong Nie, Helmut Schmid, Hinrich Schütze, Nima Mesgarani, and Jonathan Brennan. 2025a. Large language models as neurolinguistic subjects: Discrepancy between performance and competence. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19284– 19302, Vienna,… find
  19. Linyang He, Tianjun Zhong, Richard Antonello, Gavin Mischler, Micah Goldblum, and Nima Mesgarani. 2025b. Far from the shallow: Brain-predictive reasoning embedding through residual disentanglement. Preprint, arXiv:2510.22860. — [arXiv:2510.22860](https://arxiv.org/abs/2510.22860) find
  20. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
  21. Shailee Jain and Alexander Huth. 2018. Incorporating context into language encoding models for fmri. In Advances in Neural Information Processing Systems, volume 31, pages 6629–6638. Curran Associates, Inc. find
  22. Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini. 2008. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2. find
  23. Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G. Huth. 2023. A natural language fMRI dataset for voxel-wise encoding models. Scientific Data, 10(1):555. find
  24. Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Rémi King. 2022. Toward a realistic model of speech processing in the brain with self-supervised learning. In Advances in Neural Information Processing Systems, volume 35… find
  25. Gavin Mischler, Yinghao Aaron Li, Stephan Bickel, Ashesh D. Mehta, and Nima Mesgarani. 2024. Contextual feature extraction hierarchies converge in large language models and the brain. Nature Machine Intelligence, 6(12):1467–1477. find
  26. Omer Moussa, Dietrich Klakow, and Mariya Toneva. 2025. Improving semantic understanding in speech language models via brain-tuning. In The Thirteenth International Conference on Learning Representations. find
  27. Omer Moussa and Mariya Toneva. 2025. Brain-tuning improves generalizability and efficiency of brain alignment in speech models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  28. Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Q. Yamaguchi, Rieko Kubo, Shinji Nishimoto, and Yu Takagi. 2024. Unveiling multi-level and multi-modal semantic representations in the human brain using large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natu… find
  29. Anuja Negi, Subba Reddy Oota, Anwar O Nunez-Elizalde, Manish Gupta, and Fatma Deniz. 2025. Brain-informed fine-tuning for improved multilingual understanding in language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  30. Jingcheng Niu, Wenjie Lu, and Gerald Penn. 2022. Does BERT rediscover a classical NLP pipeline? In Proceedings of the 29th International Conference on Computational Linguistics (COLING), pages 3143– 3153, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  31. Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. 2024. Speech language models lack important brain-relevant semantics. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8503–8528, Bangkok, Thailand. Association for… find
  32. Subba Reddy Oota, Zijiao Chen, Manish Gupta, Bapi Raju Surampudi, Gael Jobard, Frederic Alexandre, and Xavier Hinaut. 2025. Deep neural networks and brain alignment: Brain encoding and decoding (survey). Transactions on Machine Learning Research. find
  33. Nico Policzer, Cameron Braunstein, and Mariya Toneva. 2025. The one where they brain-tune for social cognition: Multi-modal brain-tuning on friends. In NeurIPS 2025 Workshop on Foundation Models for the Brain and Body. find
  34. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–2…
  35. Long Papers), pages 3758–3769, Online. Association for Computational Linguistics. find
  36. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)… find
  37. Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. 2018. Brain-score: Which artificial neural network for object recognition is mo… find
  38. Dan Schwartz, Mariya Toneva, and Leila Wehbe. 2019. Inducing brain-relevant bias in natural language processing models. In Advances in Neural Information Processing Systems, volume 32, pages 14123–14133. Curran Associates, Inc. find
  39. Padakanti Srijith, Khushbu Pahwa, Radhika Mamidi, Bapi Raju Surampudi, Manish Gupta, and Subba Reddy Oota. 2025. Aligning text/speech representations from multimodal models with MEG brain activity during listening. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proces… find
  40. Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems, volume 32, pages 14954–14964. Curran Associates, Inc. find
  41. Nishitha Vattikonda, Aditya R. Vaidya, Richard J. Antonello, and Alexander G. Huth. 2025. Brainwavlm: Fine-tuning speech representations with brain responses to language. Preprint, arXiv:2502.08866. — [arXiv:2502.08866](https://arxiv.org/abs/2502.08866) find
  42. Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. 2014. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLOS ONE, 9(11):e112575. find
  43. Joséphine Raugel, Jérémy Rapin, Stéphane d’Ascoli, Valentin Wyart, and Jean-Remi King. 2025. Scaling and context steer LLMs along the same computational path as the human brain. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  44. Congchi Yin, Qian Yu, Zhiwei Fang, Changping Peng, and Piji Li. 2025a. Rethinking cross-subject data splitting for brain-to-text decoding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5686–5700, Suzhou, China. Association for Computational Linguist… find
  45. Yuqi Ren and Deyi Xiong. 2021. CogAlign: Learning to align textual neural representations to cognitive language processing signals. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find
  46. Zaid Zada, Samuel A. Nastase, Bobbi Aubrey, Itamar Jalon, Sebastian Michelmann, Haocheng Wang, Liat Hasenfratz, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Sasha Devore, Adeen Flinker, Orrin Devinsky, Ariel Goldstein, and Uri Hasson. 2025. The “Podcast” ECoG dataset for modeling ne… find
  47. A Data and Preprocessing A.1 ECoG Preprocessing Pipeline . . . A.2 Electrode Coverage and Selection 12 12 12 B Model Training B.1 Hyperparameter Configuration . . B.2 Computational Resources . . . . . 13 13 13 C Evaluation Details C.1 Brain Alignment Computation . . C.2 Downstream Task Specification… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-temporal-precision-matters-brain-tuning/