Elamur Research⌕
☰
home / papers / 2026 / Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

Paper · 2026 · Pedashenko Vladislav, Laida Kushnareva, Yana Khassan Nibal et al.

Summary

Intrinsic dimension (ID) of LLM embeddings is often used to study training dynamics, scaling and dataset structure. However, it was unclear which properties of a text actually drive ID. This paper connects ID to interpretable text characteristics. It first argues, formally and empirically, that ID is a separate signal from prediction-based measures. Once text length is controlled for, ID and (cross-)entropy are essentially unrelated, because ID depends on hidden-state geometry rather than on the unembedding step.

The authors estimate ID with PHD and other estimators on embeddings from Gemma, RoBERTa and Qwen across diverse corpora. They then explain the variation in three ways:

  • linguistic diagnostics, including TAACO lexical-diversity and cohesion measures and syntactic indicators;
  • sparse autoencoder (SAE) features that correlate with ID;
  • steering experiments that test whether those features cause the change.

ID separates genres consistently. Scientific text has the lowest ID, encyclopedic text sits in the middle, and fiction or opinion writing has the highest. The practical takeaway is that complexity analyses relying only on low-ID corpora, such as Wikipedia-like text, can be misleading.

Key points

  • Genre ordering holds for every model tested. Scientific prose has ID of roughly 8, encyclopedic content roughly 9, and creative or opinion writing roughly 10.5.
  • Coherent texts fall in a narrow ID band of about 5–15. Because the manifold is strongly curved, a global linear embedding needs about 60–100 dimensions, so PCA-style reduction cannot replace ID.
  • SAE features that lower ID: formal tone, research-report templates and statistics.
  • SAE features that raise ID: personalization, emotion, storytelling and forum-like conversational style. Steering confirms a causal effect most clearly for these ID-raising features.
  • Linguistic correlates: higher lexical diversity goes with higher ID, while repetition between sentences goes with lower ID. Syntactic measures carry little signal.
  • Detecting broken output: looped or repetitive generations give extremely low ID, while incoherent fragments give extremely high ID.
  • Compression: the gzip ratio constrains the plausible range of ID but does not predict it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Pedashenko Vladislav, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov and Irina Piontkovskaya. 2026. Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.370

@inproceedings{vladislav2026unveiling,
  title = {{Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story}},
  author = {Pedashenko Vladislav, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov and Irina Piontkovskaya},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.370},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.370}
}

References

44 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro…
  2. Laurent Amsaleg, Oussama Chelly, Michael E. Houle, Ken ichi Kawarabayashi, Miloš Radovanović, and Weeris Treeratanajaru. 2022. Intrinsic dimensionality estimation within tight localities: A theoretical and experimental analysis. arXiv preprint arXiv:2209.14475. — [arXiv:2209.14475](https://arxiv.org/abs/2209.14475) find
  3. Stefan Arnold. 2025. Memorization in language models through the lens of intrinsic dimension. In Proceedings of the First Workshop on Large Language Model Memorization (L2M2), pages 23–28, Vienna, Austria. Association for Computational Linguistics. find
  4. Rajendra Bhatia. 1997. Matrix Analysis, volume 1. Springer. find
  5. Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli. 2021. Intrinsic dimension, persistent homology and generalization in neural networks. Advances in neural information processing systems, 34:6776–6789. find
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  7. S. A. Crossley, K. Kyle, and M. Dascalu. 2019. The tool for the automatic analysis of cohesion 2.0: Integrating semantic similarity and text overlap. Behavioral Research Methods 51(1), pp. 14-27. find
  8. Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAID: A shared benchmark for robust evaluation of machine-generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui…
  9. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Preprint, a… find
  10. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find
  11. Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio. 2017. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7. find
  12. Yanzhu Guo, Guokan Shang, and Chloé Clavel. 2025. Benchmarking linguistic diversity of large language models. Preprint, arXiv:2412.10271.
  13. Alex Havrilla and Wenjing Liao. 2024. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. In Advances in Neural Information Processing Systems, volume 37, pages 42162–42210. find
  14. Xin He, Jiangchao Yao, Yuxin Wang, Zhenheng Tang, Ka Chun Cheung, Simon See, Bo Han, and Xiaowen Chu. 2023. Nas-lid: Efficient neural architecture search with local intrinsic dimension. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):7839–7847. find
  15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
  16. Laida Kushnareva, Tatiana Gaintseva, German Magai, Serguei Barannikov, Dmitry Abulkhanov, Kristian Kuznetsov, Eduard Tulchinskii, Irina Piontkovskaya, and Sergey Nikolenko. 2024. Ai-generated text boundary detection with roft. Preprint, arXiv:2311.08349. — [arXiv:2311.08349](https://arxiv.org/abs/2311.08349) find
  17. Kristian Kuznetsov, Laida Kushnareva, Anton Razzhigaev, Polina Druzhinina, Anastasia Voznyuk, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. 2025. Feature-level insights into artificial text detection with sparse autoencoders. In Findings of the Association for Computational Linguistic… find
  18. Kristian Kuznetsov, Eduard Tulchinskii, Laida Kushnareva, German Magai, Serguei Barannikov, Sergey Nikolenko, and Irina Piontkovskaya. 2024. Robust AI-generated text detection by restricted embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17036–17055, Miami… find
  19. Jin Hwa Lee, Thomas Jiralerspong, Lei Yu, Yoshua Bengio, and Emily Cheng. 2025. Geometric signatures of compositionality across a language model’s lifetime. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5292–5320, Vienna, Au… find
  20. Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 278–300, Miami, Florida, US. Association for Computat… find
  21. Philip M. McCarthy. 2005. An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). Ph.D. thesis, The University of Memphis. find
  22. Hariharan Narayanan and Sanjoy Mitter. 2010. Sample complexity of testing the manifold hypothesis. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc. find
  23. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024.001. find
  24. Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. 2021. The intrinsic dimension of images and its impact on learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find
  25. Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. 2024. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 868–874, St. J… find
  26. Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. find
  27. Benjamin Matthias Ruppik, Julius von Rohrscheidt, Carel van Niekerk, Michael Heck, Renato Vukovic, Shutong Feng, Hsien chin Lin, Nurul Lubis, Bastian Rieck, Marcus Zibrowius, and Milica Gašić. 2025. Less is more: Local intrinsic dimensions of contextual language models. Preprint, arXiv:2506.01034. — [arXiv:2506.01034](https://arxiv.org/abs/2506.01034) find
  28. Benjamin Schweinhart. 2021. Persistent homology and the upper box dimension. Discrete & Computational Geometry, 65(2):331–364. find
  29. Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C Wallace. 2024. Detection and measurement of syntactic templates in generated text. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6416–6431, Miami, Florida, USA. Association for Computational L…
  30. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. Preprint, arXiv:1804.08838. find
  31. Lee Sharkey, Hoagy Cunningham, Aidan Ewart, Logan Riggs, and Robert Huben. 2023. Sparse autoencoders find highly interpretable features in language models. Preprint, arXiv:2309.08600. find
  32. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Lucas Shen. 2022. Lexicalrichness: A small module to compute textual lexical richness. find
  33. Hermann H Somers. 1966. Statistical methods in literary analysis. The computer and literary style, 128:140. find
  34. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. find
  35. Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1–5. Ieee. find
  36. Hayato Tsukagoshi and Ryohei Sasano. 2025. Redundancy, isotropy, and intrinsic dimensionality of prompt-based text embeddings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25915–25930, Vienna, Austria. Association for Computational Linguistics. find
  37. Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. 2023a. Intrinsic dimension estimation for robust detection of ai-generated texts. In Advances in Neural Information Processing Systems, volume…
  38. Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. 2023b. Intrinsic dimension estimation for robust detection of AI-generated texts. In Advances in Neural Information Processing Systems. 37th N…
  39. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for C…
  40. Karthik Viswanathan, Yuri Gardinazzi, Giada Panerai, Alberto Cazzaniga, and Matteo Biagetti. 2025. The geometry of tokens in internal representations of large language models. Preprint, arXiv:2501.10573. — [arXiv:2501.10573](https://arxiv.org/abs/2501.10573) find
  41. Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, Jinyan Su, Minh Ngoc Ta, Mervat Abassy, Kareem Ashraf Elozeiri, Saad El Dine Ahmed El Etter, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar,… find
  42. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  43. Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. 2024. Characterizing truthfulness in large language model generations with local intrinsic dimension. In ICML. find
  44. Viacheslav Yusupov, Danil Maksimov, Ameliia Alaeva, Anna Vasileva, Anna Antipina, Tatyana Zaitseva, Alina Ermilova, Evgeny Burnaev, and Egor Shvetsov. 2025. From internal representations to text quality: A geometric approach to llm evaluation. Preprint, arXiv:2509.25359. — [arXiv:2509.25359](https://arxiv.org/abs/2509.25359) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vladislav-2026-unveiling-intrinsic-dimension-texts-academic/