Elamur Research⌕
☰
home / papers / 2023 / UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language

UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language

Summary

Most earlier brain-decoding work maps single word-level fMRI volumes to a small, fixed vocabulary. That is unrealistic, because fMRI samples the brain slowly and each frame overlaps several words. The authors define fMRI2text, a new open-vocabulary task: turn a sequence of fMRI frames, recorded while people read or listen to natural text or speech, into the corresponding token sequence. As a baseline they propose UniCoRN, an encoder–decoder framework meant to work for different cognitive signals.

Training has three stages:

  1. A CNN encoder learns to rebuild individual fMRI volumes ("snapshots").
  2. A Transformer encoder learns to rebuild sequences of those snapshot embeddings ("series").
  3. The resulting signal embeddings are passed through a projection layer into the pre-trained BART model, which generates text.

UniCoRN reaches 34.77% BLEU-4 on fMRI2text. Applied to EEG2text, it gets 37.04% BLEU-4, beating the earlier Wang and Ji (2022) baseline. The work suggests that continuous fMRI streams under naturalistic conditions can be decoded into coherent text, and that one shared architecture can handle both fMRI and EEG.

Key points

  • New task. fMRI2text decodes a fixed-length window of T consecutive fMRI frames, sampled every repetition time (TR), into open-vocabulary text. Each frame aligns with a variable number of tokens, not exactly one word.
  • Architecture.
    • Phase 1: a CNN snapshot encoder/decoder, similar to Malkiel et al. (2021). The decoder is deliberately simpler than the encoder.
    • Phase 2: a multi-layer Transformer series encoder.
    • Phase 3: decoding with BART.
    • Both reconstruction phases use an MAE loss.
  • Main results. fMRI2text: BLEU-1 62.90, BLEU-4 34.77, ROUGE-1 F 59.52. EEG2text: BLEU-4 37.04 versus 34.65 for the prior baseline, and ROUGE-1 F 64.39 versus 58.78.
  • fMRI ablation. Removing both reconstruction phases drops BLEU-4 to 19.68. Series reconstruction helps somewhat more than snapshot reconstruction.
  • EEG ablation. Using snapshot reconstruction alone hurts EEG results (BLEU-4 25.28), but combining both phases gives the best score.
  • Series length. Increasing the series length T helps when T is small, then the gains level off or reverse. The authors attribute this to Transformer limits on long-range dependencies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.741 · arXiv 2307.05355

@inproceedings{xi2023unicorn,
  title = {{UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language}},
  author = {Xi, Nuwa and Zhao, Sendong and Wang, Haochun and Liu, Chi and Qin, Bing and Liu, Ting},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.741},
  eprint = {2307.05355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.741}
}

References

39 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicolas Affolter, Beni Egressy, Damian Pascual, and Roger Wattenhofer. 2020. Brain2word: decoding brain activity for language generation. arXiv preprint arXiv:2009.04765. — [arXiv:2009.04765](https://arxiv.org/abs/2009.04765) find
  2. Hassan Akbari, Bahar Khalighinejad, Jose L Herrero, Ashesh D Mehta, and Nima Mesgarani. 2019. Towards reconstructing intelligible speech from the human auditory cortex. Scientific reports, 9(1):874. find
  3. Shohini Bhattasali, Murielle Fabre, Wen-Ming Luh, Hazem Al Saied, Mathieu Constant, Christophe Pallier, Jonathan R Brennan, R Nathan Spreng, and John Hale. 2019. Localising memory retrieval and syntactic composition: an fmri study of naturalistic language comprehension. Language, Cognition and Neuro… find
  4. Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2767–2796. find
  5. Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. 2022. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.06956. — [arXiv:2211.06956](https://arxiv.org/abs/2211.06956) find
  6. Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. 2022. Decoding speech from non-invasive brain recordings. arXiv preprint arXiv:2208.12266. — [arXiv:2208.12266](https://arxiv.org/abs/2208.12266) find
  7. Bing Du, Xiaomu Cheng, Yiping Duan, and Huansheng Ning. 2022. fmri brain decoding and its applications in brain–computer interface: A survey. Brain Sciences, 12(2):228. find
  8. Liberty S Hamilton and Alexander G Huth. 2020. The revolution will not be controlled: natural stimuli in speech neuroscience. Language, cognition and neuroscience, 35(5):573–582. find
  9. Nora Hollenstein, Jonathan Rotsztejn, Marius Troen-dle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading. Scientific data, 5(1):1–13. find
  10. Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453– 458. find
  11. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find our card
  12. Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. 2020. Very deep transformers for neural machine translation. arXiv preprint arXiv:2008.07772. — [arXiv:2008.07772](https://arxiv.org/abs/2008.07772) find
  13. Itzik Malkiel, Gony Rosenman, Lior Wolf, and Talma Hendler. 2021. Pre-training and fine-tuning transformers for fmri prediction tasks. arXiv preprint arXiv:2112.05761. — [arXiv:2112.05761](https://arxiv.org/abs/2112.05761) find
  14. Radek Martinek, Martina Ladrova, Michaela Sidikova, Rene Jaros, Khosrow Behbehani, Radana Kahankova, and Aleksandra Kawala-Sterniuk. 2021. Advanced bioelectrical signal processing methods: Past, present and future approach—part ii: Brain signals. Sensors, 21(19):6343. find
  15. Sean L Metzger, Jessie R Liu, David A Moses, Maximilian E Dougherty, Margaret P Seaton, Kaylo T Little-john, Josh Chartier, Gopala K Anumanchipalli, Ade-lyn Tu-Chan, Karunesh Ganguly, et al. 2022. Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paral… find
  16. Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. science, 320(5880):1191–1195. find
  17. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32. find our card
  18. Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):1–13. find
  19. Kevin M Pitt and Aimee Dietz. 2022. Applying implementation science to support active collaboration in noninvasive brain–computer interface development and translation for augmentative and alternative communication. American Journal of Speech-Language Pathology, 31(1):515–526. find
  20. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267. find our card
  21. Adrien B Rapeaux and Timothy G Constandinou. 2021. Implantable brain machine interfaces: first-in-human studies, technology challenges and trends. Current opinion in biotechnology, 72:102–111. find
  22. Shiv Kumar Mudgal, Suresh K Sharma, Jitender Chaturvedi, and Anil Sharma. 2020. Brain computer interface advancement in neurosciences: Applications and issues. Interdisciplinary Neurosurgery, 20:100694. find
  23. Sergio Ruiz, Korhan Buyukturkoglu, Mohit Rana, Niels Birbaumer, and Ranganatha Sitaram. 2014. Real-time fmri brain computer interfaces: self-regulation of single brain regions to networks. Biological psychology, 95:4–20. find
  24. Samuel A Nastase, Ariel Goldstein, and Uri Hasson. 2020. Keep it real: rethinking the primacy of experimental control in cognitive neuroscience. NeuroImage, 222:117254. find
  25. Simanto Saha, Khondaker A Mamun, Khawza Ahmed, Raqibul Mostafa, Ganesh R Naik, Sam Darvishi, Ahsan H Khandoker, and Mathias Baumert. 2021. Progress in brain computer interface: Challenges and opportunities. Frontiers in Systems Neuroscience, 15:578875. find
  26. Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. 2021. The “narratives” fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1):1–22. find
  27. Uta Noppeney and Catherine J Price. 2004. An fmri study of syntactic adaptation. Journal of Cognitive Neuroscience, 16(4):702–713. find
  28. Mark Pagel. 2017. Q&a: What is human language, when did it evolve and why should we care? BMC biology, 15(1):1–6. find
  29. Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. 2009. Zero-shot learning with semantic output codes. Advances in neural information processing systems, 22. find
  30. Jerrin Thomas Panachakel and Angarai Ganesan Ramakrishnan. 2021. Decoding covert speech from eeg-a comprehensive review. Frontiers in Neuroscience, page 392. find
  31. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631… find
  32. Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019. Towards sentence-level brain decoding with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7047–7054. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find
  34. Athena Vouloumanos, Kent A Kiehl, Janet F Werker, and Peter F Liddle. 2001. Detection of sounds in the auditory stream: event-related fmri evidence for differential activation to speech and nonspeech. Journal of Cognitive Neuroscience, 13(7):994–1005. find
  35. Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. 2020. Fine-grained neural decoding with distributed word representations. Information Sciences, 507:256–272. find
  36. Zhenhailong Wang and Heng Ji. 2022. Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5350–5358. find
  37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language p… find our card
  38. Jonathan R Wolpaw. 2007. Brain-computer interfaces (bcis) for communication and control. In Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, pages 1–2. find
  39. Shuxian Zou, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2021. Towards brain-to-text generation: Neural decoding with pre-trained encoder-decoder models. In NeurIPS 2021 AI for Science Workshop. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/xi-2023-unicorn-unified-cognitive-signal-reconstruction/