Elamur Research⌕
☰
home / papers / 2025 / Layer by Layer: Uncovering Hidden Representations in Language Models

Layer by Layer: Uncovering Hidden Representations in Language Models

Paper · 2025 · Oscar Skean, Md Rifat Arefin, Dan Zhao et al.

Summary

Most pipelines built on large language models take embeddings from the last layer, on the assumption that earlier layers hold only shallow cues. Skean et al. test this assumption with a layer-by-layer study. They cover transformers, state-space models and encoders such as BERT, at sizes from tens of millions to billions of parameters, and they extend the analysis to vision models. They treat the output of every layer as an embedding and evaluate it across 32 tasks drawn from MTEB, the Massive Text Embedding Benchmark. To explain the results, the authors combine information-theoretic, geometric and invariance-based quality measures into one framework, linked through matrix-based entropy computed from the eigenvalues of the Gram matrix. Mid-depth layers repeatedly beat the final one, by up to 16% in downstream accuracy. The last layer tends to become too tuned to the pretraining objective. Autoregressive models show a clear entropy dip in their middle layers, and an autoregressive image model shows the same dip. This suggests that the training objective, not the data modality, drives the effect. Intermediate embeddings therefore look like an underused source of features.

Key points

  • Evaluation spans 32 MTEB tasks. Example models include Pythia-410m, Mamba-370m and BERT-base, with layers compared by relative depth percentage. Intermediate layers outperform the final layer by as much as 16%.
  • The framework rests on matrix-based entropy of order α (α = 1 by default). It links compression, curvature and invariance metrics such as InfoNCE, LiDAR and DiME. DiME, curvature and InfoNCE track downstream performance well.
  • Autoregressive models show a pronounced mid-layer "compression valley." Masked or bidirectional models change more mildly across depth. Chain-of-thought finetuning reshapes mid-layer entropy.
  • Among vision models, only the autoregressive AIM peaks in accuracy at intermediate layers. The others, including BEIT, improve steadily toward the final layer. AIMv2 shows an entropy valley without the mid-layer accuracy gain.
  • The paper appeared at ICML 2025. Code is available at github.com/OFSkean/information_flow.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun and Ravid Shwartz-Ziv. 2025. Layer by Layer: Uncovering Hidden Representations in Language Models. arXiv:2502.02013.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.02013 · arXiv 2502.02013

@misc{skean2025layer,
  title = {{Layer by Layer: Uncovering Hidden Representations in Language Models}},
  author = {Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun and Ravid Shwartz-Ziv},
  year = {2025},
  doi = {10.48550/arxiv.2502.02013},
  eprint = {2502.02013},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.02013}
}

References

64 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, K. K., Mondal, A. K., Ghosh, A., and Richards, B. α-ReQ: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. NeurIPs, 2022. find
  2. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. ICML, 2020. find
  3. Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. ICLR, 2017. find
  4. Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, J., Laio, A., and Baroni, M. Emergence of a high-dimensional abstraction phase in language transformers. ICLR, 2025. find
  5. Arefin, M. R., Subbaraj, G., Gontier, N., LeCun, Y., Rish, I., Shwartz-Ziv, R., and Pal, C. Seq-VCR: Preventing collapse in intermediate transformer representations for enhanced reasoning. ICLR, 2025. find
  6. Csordás, R., Manning, C. D., and Potts, C. Do language models use their depth efficiently? arXiv, 2025. find
  7. Bach, F. Information theory with kernel methods. IEEE Transactions on Information Theory, 2022. find
  8. DeepSeek-AI. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv, 2025.
  9. Bao, H., Dong, L., Piao, S., and Wei, F. BeIT: Bert pre-training of image transformers. ICLR, 2022. find
  10. Deletang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., et al. Language modeling is compression. ICLR, 2024.
  11. Barbero, F., Arroyo, A., Gu, X., Perivolaropoulos, C., Bronstein, M., Veličković, P., and Pascanu, R. Why do LLMs attend to the first token? arXiv, 2025. find
  12. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
  13. Jin, M., Yu, Q., Huang, J., Zeng, Q., Wang, Z., Hua, W., Zhao, H., Mei, K., Meng, Y., Ding, K., et al. Exploring concept depth: How large language models acquire knowledge at different layers? arXiv, 2024. find
  14. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
  15. Lad, V., Gurnee, W., and Tegmark, M. The remarkable robustness of LLMs: Stages of inference? arXiv, 2024. find
  16. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alphacode. Science, 2022.
  17. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv, 2024.
  18. Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. Linguistic knowledge and transferability of contextual representations. NAACL, 2019.
  19. El-Nouby, A., Klein, M., Zhai, S., Bautista, M. A., Toshev, A., Shankar, V., Susskind, J. M., and Joulin, A. Scalable pre-training of large autoregressive image models. ICML, 2024. find
  20. Ma, E. NLP Augmentation, 2019. URL https://github.com/makcedward/nlpaug. ↗
  21. Fan, S., Jiang, X., Li, X., Meng, X., Han, P., Shang, S., Sun, A., Wang, Y., and Wang, Z. Not all layers of LLMs are necessary during inference. arXiv, 2024. find
  22. Mallen, A. T. and Belrose, N. Eliciting latent knowledge from quirky language models. ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. find
  23. Fini, E., Shukor, M., Li, X., Dufter, P., Klein, M., Haldimann, D., Aitharaju, S., da Costa, V. G. T., Béthune, L., Gan, Z., et al. Multimodal autoregressive pre-training of large vision encoders. CVPR, 2025. find
  24. Mamou, J., Le, H., Del Rio, M. A., Stephenson, C., Tang, H., Kim, Y., and Chung, S. Emergence of separable manifolds in deep language representations. ICML, 2020. find
  25. Garrido, Q., Balestriero, R., Najman, L., and Lecun, Y. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. ICML, 2023. find
  26. Marion, P., Wu, Y.-H., Sander, M. E., and Biau, G. Implicit regularization of deep residual networks towards neural odes. ICLR, 2024. find
  27. Giraldo, L. G. S., Rao, M., and Principe, J. C. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory, 2014. find
  28. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. ICLR, 2017.
  29. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. COLM, 2024. find
  30. Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive text embedding benchmark. EACL, 2022.
  31. Gu, X., Pang, T., Du, C., Liu, Q., Zhang, F., Du, C., Wang, Y., and Lin, M. When attention sink emerges in language models: An empirical view. ICLR, 2025. find
  32. Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. ICLR, 2018.
  33. Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv, 2023. find
  34. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
  35. Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. Training large language models to reason in a continuous latent space. arXiv, 2024. find
  36. Park, K., Choe, Y. J., Jiang, Y., and Veitch, V. The geometry of categorical and hierarchical concepts in large language models. ICML 2024 Workshop on Mechanistic Interpretability, 2024a. find
  37. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. CVPR, 2022.
  38. Hosseini, E. and Fedorenko, E. Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language. NeurIPs, 2023. find
  39. Park, K., Choe, Y. J., and Veitch, V. The linear representation hypothesis and the geometry of large language models. ICML, 2024b.
  40. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. ICML, 2021.
  41. Thilak, V., Huang, C., Saremi, O., Dinh, L., Goh, H., Nakkiran, P., Susskind, J. M., and Littwin, E. LiDAR: Sensing linear probing performance in joint embedding ssl architectures. ICLR, 2024. find
  42. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. NeurIPs, 2017. find
  43. Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. ECCV, 2020. find
  44. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.
  45. Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Oseledets, I., Dimitrov, D., and Kuznetsov, A. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. EACL, 2024. find
  46. Valeriani, L., Doimo, D., Cuturello, F., Laio, A., Ansuini, A., and Cazzaniga, A. The geometry of hidden representations of large transformer models. NeurIPs, 2023. find
  47. Rényi, A. On measures of entropy and information. Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, 1961. find
  48. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. NeurIPs, 2017.
  49. Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. European signal processing conference, 2007. find
  50. Voita, E., Sennrich, R., and Titov, I. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. EMNLP-IJCNLP, 2019.
  51. Saponati, M., Sager, P., Aceituno, P. V., Stadelmann, T., and Grewe, B. The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in transformer training. arXiv, 2025. find
  52. Wei, L., Tan, Z., Li, C., Wang, J., and Huang, W. Diff-eRank: A novel rank-based metric for evaluating large language models. NeurIPs, 2024. find
  53. Scholkopf, B. and Smola, A. J. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2018. find
  54. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. ICLR, 2024. find
  55. Shwartz-Ziv, R. Information flow in deep neural networks. PhD thesis, Hebrew University, 2022. find
  56. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., R… find
  57. Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. Entropy, 2019. find
  58. Shwartz-Ziv, R., Balestriero, R., Kawaguchi, K., Rudner, T. G., and LeCun, Y. An information theory perspective on variance-invariance-covariance regularization. NeurIPs, 2023. find
  59. Zhao, Z., Ziser, Y., and Cohen, S. B. Layer by layer: Uncovering where multi-task learning happens in instruction-tuned large language models. EMNLP-IJCNLP, 2024. find
  60. Skean, O., Osorio, J. K. H., Brockmeier, A. J., and Giraldo, L. G. S. DiME: Maximizing mutual information by a difference of matrix-based entropies. arXiv, 2023. find
  61. Zhouyin, Z. and Liu, D. Understanding neural networks with logarithm determinant entropy estimator. arXiv, 2021. find
  62. Skean, O., Dhakal, A., Jacobs, N., and Giraldo, L. G. S. FroSSL: Frobenius norm minimization for self-supervised learning. ECCV, 2024. find
  63. Sorscher, B., Ganguli, S., and Sompolinsky, H. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 2022. find
  64. Tenney, I., Das, D., and Pavlick, E. BERT rediscovers the classical nlp pipeline. NAACL, 2019.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/skean-2025-layer-layer-uncovering-hidden-representations/