Elamur Research⌕
☰
home / papers / 2025 / Do Large Language Models Mirror Cognitive Language Processing?

Do Large Language Models Mirror Cognitive Language Processing?

Paper · 2025 · Yuqi Ren, Renren Jin, Tongxuan Zhang et al.

Summary

This paper asks whether the internal text representations of modern large language models resemble the way the human brain processes language, and which training choices strengthen or weaken that resemblance. The authors compare sentence embeddings from 23 open-source LLMs (Amber, LLaMA, LLaMA2, LLaMA3, Vicuna, Mistral, Zephyr and their tuned variants, from 7B to 70B) with fMRI recordings of people reading the same English stimulus sentences. They use Representational Similarity Analysis (RSA). For both brain data and model embeddings, they build a matrix of pairwise sentence dissimilarities, then measure how closely the two matrices agree. They test several factors: pre-training data size, parameter count, alignment training (SFT, RLHF, DPO), prompt type, sentiment polarity and layer depth. Models score higher on this LLM-brain similarity when they are larger, trained on more data, or alignment-tuned. The score also tracks standard benchmark results. This suggests brain alignment could serve as an extra way to evaluate LLMs and to interpret them through neuroscience.

Key points

  • Method. RSA compares two representational dissimilarity matrices, each built from 1 − Pearson ρ between sentence pairs. Agreement between the brain matrix and the model matrix is scored with Pearson, Spearman, Cosine and Euclidean measures.
  • Training effects. Alignment training consistently raises similarity. With intermediate-layer embeddings, LLaMA2-70B scores 0.2776 (Pearson) and LLaMA2-70B-chat scores 0.3188. LLaMA3-70B-instruct reaches the top score of 0.3377, while the weakest model, the pre-trained Amber, scores 0.2121.
  • Prompts. Three input conditions were compared: no prompt, an explicit instruction ("Please complete the following text:"), and a noisy prefix of 5 random English words. Explicit instructions increase similarity, while the meaningless prefix tends to lower it.
  • Sentiment. Using 10 positive and 10 negative concept words, the authors find that models resemble brain responses more closely for positive words.
  • Benchmarks and limitations. Similarity correlates positively with benchmarks such as MMLU and Chatbot Arena. LLaMA3-70B-instruct is an outlier: it is strong on alignment evaluations but further from brain data than the trend predicts. The study covers only open-source models and English-only fMRI stimuli.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tianjin UniversityTianjin · China41 works · 1,128 citations🎓Tianjin Normal UniversityTianjin · China2 works · 11 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yuqi Ren, Renren Jin, Tongxuan Zhang and Deyi Xiong. 2025. Do Large Language Models Mirror Cognitive Language Processing?. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ren2025do,
  title = {{Do Large Language Models Mirror Cognitive Language Processing?}},
  author = {Yuqi Ren, Renren Jin, Tongxuan Zhang and Deyi Xiong},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.201/}
}

References

50 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mostafa Abdou, Artur Kulmizev, Felix Hill, Daniel M Low, and Anders Søgaard. 2019. Higher-order comparisons of sentence encoder representations. arXiv preprint arXiv:1909.00303. find
  2. Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019. Blackbox meets blackbox: Representational similarity and stability analysis of neural language models and brains. arXiv preprint arXiv:1906.01539. — [arXiv:1906.01539](https://arxiv.org/abs/1906.01539) find
  3. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly. find
  4. Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. arXiv preprint arXiv:1905.06401. find
  5. Andrew C Connolly, J Swaroop Guntupalli, Jason Gors, Michael Hanke, Yaroslav O Halchenko, Yu-Chien Wu, Hervé Abdi, and James V Haxby. 2012. The representation of biological classes in the human brain. Journal of Neuroscience, 32(8):2608–2618. find
  6. Richard J Davidson. 2003. Affective neuroscience and psychophysiology: Toward a synthesis. Psychophysiology, 40(5):655–665. find
  7. Sophie Denève, Alireza Alemi, and Ralph Bourdoukan. 2017. The brain as an efficient and robust adaptive learner. Neuron, 94(5):969–977. find
  8. Jörn Diedrichsen and Nikolaus Kriegeskorte. 2017. Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis. PLoS computational biology, 13(4):e1005508. find
  9. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023,, pages…
  10. Kshitij Dwivedi and Gemma Roig. 2019. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12387–12396. find
  11. Angela D Friederici. 2011. The brain basis of language processing: from structure to function. Physiological reviews. find
  12. Paul Friedl. 2023. Dis/similarities in the design and development of legal and algorithmic normative systems: the case of perspective api. Law, Innovation and Technology, 15(1):25–59. find
  13. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459.
  14. SK Gargee, Pranav Bhargav Gopinath, Shridhar Reddy SR Kancharla, CR Anand, and Anoop S Babu. 2022. Analyzing and addressing the difference in toxicity prediction between different comments with same semantic meaning in google’s perspective api. In ICT Systems and Sustainability: Proceedings of ICT4S… find
  15. Xiaochuang Han. 2023. In-context alignment: Chat with vanilla language models before fine-tuning. arXiv preprint arXiv:2308.04275. — [arXiv:2308.04275](https://arxiv.org/abs/2308.04275) find
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021.
  17. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  18. Nora Hollenstein, Antonio de la Torre, Nicolas Langer, and Ce Zhang. 2019. Cognival: A framework for cognitive word embedding evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, pages 538–549. find
  19. Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. 2023. Emotionally numb or empathetic? evaluating how LLMs feel using emotionbench. arXiv preprint arXiv:2308.03656. — [arXiv:2308.03656](https://arxiv.org/abs/2308.03656) find
  20. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics.
  21. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  22. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  23. Marcie L. King, Iris I. A. Groen, Adam Steel, Dwight J. Kravitz, and Chris I. Baker. 2019. Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images. NeuroImage, 197:368–382. find
  24. Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, page 4. find
  25. Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3758–3769. find
  26. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. find
  27. Yuqi Ren and Deyi Xiong. 2022. Bridging between cognitive processing signals and linguistic features via a unified attentional network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 49–58. find
  28. Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Z… find
  29. Yifei Luo, Minghui Xu, and Deyi Xiong. 2022. Cog-taskonomy: Cognitively inspired task taxonomy is beneficial to transfer learning in NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 904–920. find
  30. Meta. 2024. Introducing meta LLaMA 3: The most capable openly available llm to date. web link. find
  31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26.
  32. Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008. Predicting human brain activity associated with the meanings of nouns. science, 320(5880):1191–1195. find
  33. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e… find
  34. Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019. Towards sentence-level brain decoding with distributed representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7047–7054. find
  35. Michel Thiebaut de Schotten and Stephanie J Forkel. 2022. The emergent properties of the connected brain. Science, 378(6619):505–510. find
  36. Mariya Toneva and Leila Wehbe. 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 149… find
  37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  38. OpenAI. 2023. Introducing chatgpt. web link. find
  39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  40. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  41. Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. 2018. Toward a universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):963. find
  42. Yuqi Ren and Deyi Xiong. 2021. Cogalign: Learning to align textual neural representations to cognitive language processing signals. In Proceedings of the 59th Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine F… find
  43. Xuena Wang, Xueting Li, Zi Yin, Yue Wu, and Jia Liu. 2023. Emotional intelligence of large language models. Journal of Pacific Rim Psychology, 17:18344909231213958. find
  44. Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. 2014. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 233–243. find
  45. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  46. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
  47. Haoyan Xu, Brian Murphy, and Alona Fyshe. 2016. Brainbench: A brain-image test suite for distributional semantic models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2017–2021. The Association for… find
  48. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  49. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023a. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685.
  50. Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipen… — Secrets of RLHF in Large Language Models Part I: PPO

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ren-2025-do-large-language-models-mirror/