Elamur Research⌕
☰
home / papers / 2026 / Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs

Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs

Paper · 2026 · Yiheng Yang, Yujie Wang, Chi Ma et al.

Summary

Dense LLMs spend the same compute on every token, however easy or hard it is. Existing dynamic-activation methods each have a weakness. DejaVu does not generalize beyond ReLU models, Threshold Truncation (TT) costs too much overhead to give real speedups, and Griffin loses noticeable accuracy. The authors propose CLADA, a training-free framework inspired by two ERP components of human language processing: N400 (prediction) and P600 (reanalysis). They first show, through panel regressions on XSum-derived sequences and LLaMA-3-8B activations, that activation patterns depend strongly on the sequence prefix ("statistical sparsity"). They also show that these patterns shift with token-level surprisal and entropy ("semantic adaptability"). CLADA uses this in two steps. Offline-searched, error-controlled layer-wise thresholds give a sparse baseline, and real-time cognitive-load signals then adjust it. Across six LLMs and nine benchmarks, it delivers about 20% average speedup with under 2% accuracy loss, without retraining or changing the architecture. This makes it a practical option for cheaper inference and links psycholinguistics to LLM efficiency.

Key points

  • The baseline thresholds guarantee more than 40% sparsity. Surprisal and entropy then modulate them per token. Griffin degrades by over 5%, and TT gives almost no speedup.
  • On LLaMA2-7B, MMLU drops only from 45.83 to 44.83 with CLADA. With DejaVu, it falls to 27.02.
  • The validation corpus was built from XSum. It contains 2,000 natural and 2,000 POS-preserving random-token sequences, expanded into 36,000 hybrid sequences. Similarity was measured with CKA on LLaMA-3-8B layer 15.
  • The prefix-length effect is strong: β = 4.12 for random sequences and 3.91 for natural text. Adding surprisal and entropy raises the adjusted R² to 0.17.
  • Limitations:
    • At 32K context, CLADA beats Griffin by 1.57–1.91 s.
    • Prefill takes 10–15% of inference time for prefixes over 2048 tokens.
    • Storing activation masks raises GPU memory use for very large models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🏢MeituanBeijing · China17 works · 3,769 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yiheng Yang, Yujie Wang, Chi Ma, Lei Yu, Emmanuele Chersoni and Chu-Ren Huang. 2026. Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.270

@inproceedings{yang2026sparse,
  title = {{Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs}},
  author = {Yiheng Yang, Yujie Wang, Chi Ma, Lei Yu, Emmanuele Chersoni and Chu-Ren Huang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.270},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.270}
}

References

51 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations. find
  2. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. PIQA: Reasoning about Physical Commonsense in Natural Language. Preprint, arXiv:1911.11641.
  3. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. Preprint, arXiv:1803.05457.
  6. Harry Dong, Beidi Chen, and Yuejie Chi. 2024. Prompt-prompted adaptive structured pruning for efficient LLM generation. In First Conference on Language Modeling. find
  7. Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. 2022. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503:92–108. find
  8. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR). find
  9. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR. find
  10. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  11. Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  12. Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. 2024. Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21158–21166, Miami, Florida, USA. Ass… find
  13. Peter Hagoort. 2003. Interplay between syntax and semantics during sentence comprehension: Erp effects of combining syntactic and semantic violations. Journal of Cognitive Neuroscience, 15(6):883–899. find
  14. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  15. John Hale. 2016. Information-theoretical complexity metrics. Language and Linguistics Compass, 10(9):397–412. find
  16. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 3519–3529.
  17. Gina R Kuperberg. 2007. Neural mechanisms of language comprehension: Challenges to syntax. Brain Research, 1146:23–49. find
  18. Gina R Kuperberg, Trevor Brothers, and Edward W Wlotko. 2020. A tale of two positivities and the N400: Distinct neural signatures are evoked by confirmed and violated predictions at different levels of representation. Journal of Cognitive Neuroscience, 32(1):12–35. find
  19. Marta Kutas and Kara D Federmeier. 2011. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). Annual Review of Psychology, 62(1):621–647. find
  20. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  21. Jiaxuan Li and Richard Futrell. 2024. Decomposition of surprisal: Unified computational model of erp components in language processing. Preprint, arXiv:2409.06803. — [arXiv:2409.06803](https://arxiv.org/abs/2409.06803) find
  22. Ziang Liu, Genggeng Zhou, Jeff He, Tobia Marcucci, Li Fei-Fei, Jiajun Wu, and Yunzhu Li. 2023a. Model-based control with sparse neural dynamics. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. find
  23. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. 2023b. Deja vu: Contextual sparsity for efficient LLMs at inference time. In Proceedings of the 40th International Conference on Machine Learning, vol… find
  24. Chi Ma, Mincong Huang, Chao Wang, Yujie Wang, and Lei Yu. 2024. Dynamic Activation Pitfalls in LLaMA Models: An Empirical Study. Preprint, arXiv:2405.09274. — [arXiv:2405.09274](https://arxiv.org/abs/2405.09274) find
  25. Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. 2020. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pages 6682–6691. PMLR. find
  26. James A Michaelov, Megan D Bardolph, Cyma K Van Petten, Benjamin K Bergen, and Seana Coulson. 2024. Strong prediction: Language model surprisal explains multiple N400 effects. Neurobiology of Language, 5(1):107–135. find
  27. Ramesh Nallapati, Bowen Zhou, Cicero Dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of the 20th SIGNLL conference on computational natural language learning, pages 280–290.
  28. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As…
  29. Byung-Doh Oh, Shisen Yue, and William Schuler. 2024. Frequency Explains the Inverse Correlation of Large Language Models’ Size, Training Data Amount, and Surprisal’s Fit to Reading Times. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (… find
  30. Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. 2024. Dense training, sparse inference: Rethinking training of mixture-of-experts language models. Preprint, arXiv:2404.05567. find
  31. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
  32. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95. find
  33. Lavinia Salicchi and Yu-Yin Hsu. 2025. Not Every Metric is Equal: Cognitive Models for Predicting N400 and P600 Components During Reading Comprehension. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3648– 3654, Abu Dhabi, UAE. Association for Computational L… find
  34. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1200…
  35. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. Layer by layer: Uncovering hidden representations in language models. In Forty-second International Conference on Machine Learning. find
  36. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A simple and effective pruning approach for large language models. Preprint, arXiv:2306.11695. find
  37. Filip Szatkowski, Bartosz Wójcik, Mikoł aj Piórczyński, and Simone Scardapane. 2024. Exploiting activation sparsity with dense to dynamic-k mixture-of-experts conversion. In Advances in Neural Information Processing Systems, volume 37, pages 43245–43273. Curran Associates, Inc. find
  38. Ethan G Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470.
  39. Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. 2024. LLM Inference Unveiled: Survey and Roofline Model Insights. Preprint, arXiv:2402.16363. find
  40. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. find
  41. Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. MoEfication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022, pages 877–890, Dublin, Ireland. Association for Computational Linguistics.
  42. Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. 2024. Learn to be efficient: Build structured sparsity in large language models. Advances in Neural Information Processing Systems, 37:101969– 101991. find
  43. Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. 2024. Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training. Preprint, arXiv:2405.03133. find
  44. Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. 2024. LLaMA-MoE: Building mixture-of-experts from LLaMA with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15913–15923, Miami, Florida, US…
  45. A MoEfication (Zhang et al., 2022) dynamically routes inputs to expert subnets, but requires costly expert training and introduces routing overhead. find
  46. DS-MoE (Pan et al., 2024) introduces a framework that employs dense computation during training and switches to sparse computation during inference. find
  47. LLaMA-MoE (Zhu et al., 2024) offers a new lightweight method to transform FFNs into MoEs. find
  48. LTE (Zheng et al., 2024) achieves a superior balance between sparsity and performance by activating fewer neurons and is applicable to models with both ReLU and non-ReLU activation functions. find
  49. Lory (Zhong et al., 2024) retains the autoregressive properties of language models by adopting a causally segmented routing strategy and a similarity-based data batching method. This enables efficient expert merging operations and promotes specialization among experts in processing similar documents… find
  50. Training-Free Methods Griffin (Dong et al., 2024) uses sequence-level activation clustering (flocking) to skip redundant computations only in generation phase. Despite its simplicity, Griffin suffers from significant performance drops (\>3% on QA tasks) due to heuristic threshold selection. find
  51. B Details on Bridging Cognitive Load and Activation B.1 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-sparse-brains-are-also-adaptive/