Elamur Research⌕
☰
home / papers / 2026 / Rethinking the Relationship between the Power Law and Hierarchical Structures

Rethinking the Relationship between the Power Law and Hierarchical Structures

Summary

Across many corpora, the statistical dependence between linguistic units (phones, characters, words) weakens with distance following a power law rather than an exponential curve. Since Lin and Tegmark (2017), this pattern has often been taken as a sign of hidden hierarchical structure, and the same reasoning has been applied to child speech, birdsong and chimpanzee action sequences. Their derivation depends on two conditions. First, correlation must fall off exponentially with distance inside the tree. Second, trees must be balanced, so that surface distance grows exponentially with tree distance.

Nakaishi et al. test these conditions directly on syntactic parse trees from English (BLLIP, WikiText) and Japanese (NPCMJ). They measure mutual information along tree paths and deviations from PCFG behaviour. They also compute the same statistics for a PCFG fitted to BLLIP. Neither condition holds for real syntax. The fitted PCFG also produces trees too small to reach its theoretical asymptotic regime. The hierarchy-based explanation of power-law correlations therefore needs rethinking, possibly at the discourse level or through critical phenomena.

Key points

  • The authors pose three questions: whether correlation decays exponentially in syntactic trees, whether sequential distance grows exponentially with structural distance, and whether real trees deviate from PCFGs. Real syntax contradicts the assumed answer to every one. Mutual information within trees itself follows a power law with structural distance. Strong branching bias makes sequential distance grow more slowly than linearly. Natural trees differ significantly from PCFG-generated ones.
  • The results are consistent across the BLLIP treebank under several annotation schemes, the WikiText corpus and the Japanese treebank NPCMJ.
  • In the maximum-likelihood PCFG estimated from BLLIP, asymptotic behaviour does not appear within single sequences. This casts doubt on the argument for other sequences of sentence-like length, such as child speech and animal behaviour. It also limits how far PCFG-based synthetic data can explain how large language models learn natural syntax.
  • Future work needs large discourse-annotated corpora to estimate quantities such as I_POS, I_tag and J beyond the sentence level. An alternative account is critical phenomena, which Nakaishi et al. (2024) observed in LLM text as the temperature parameter varies.
  • The paper appeared in TACL, vol. 14, pp. 1917–1934 (2026), DOI 10.1162/TACL.a.785.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🎓The University of TokyoTokyo · Japan31 works · 7 442 citations🎓Georgetown UniversityWashington · United States62 works · 4 229 citations🔬National Institute of InformaticsTokyo · Japan12 works · 2 482 citations🔬RIKENWako · Japan6 works · 207 citations🔬National Institute for Japanese Language and LinguisticsTokyo · Japan4 works · 2 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.785

@article{nakaishi2026rethinking,
  title = {{Rethinking the Relationship between the Power Law and Hierarchical Structures}},
  author = {Nakaishi, Kai and Yoshida, Ryo and Kajikawa, Kohei and Hukuahima, Koji and Oseki, Yohei},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.785},
  url = {https://doi.org/10.1162/tacl.a.785}
}

References

65 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hirotugu Akaike. 1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6):716–723. find
  2. In Advances in Neural Information Processing Systems 37. find
  3. Anne Chao and Tsung-Jen Shen. 2003. Non-parametric estimation of shannon’s index of diversity when there are unseen species in sample. Environmental and ecological statistics, 10(4):429–443. find
  4. Eugene Charniak. 1997. Statistical techniques for natural language parsing. The AI Magazine, 18:33–44. find
  5. Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987-89 WSJCorpus Release 1 LDC2000T43. Linguistic Data Consortium. find
  6. Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on information theory, 2(3):113–124. find
  7. Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of language models: Part 1, learning hierarchical language structures. Available at SSRN. find
  8. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. find our card
  9. Eduardo G. Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti. 2012. On the origin of long-range correlations in texts. Proceedings of the National Academy of Sciences, 109(29):11582–11587. find
  10. Matthew S. Dryer. 1992. The greenbergian word order correlations. Language, 68(1):81–138. find
  11. Enrique Alvarez-Lacalle, Beate Dorow, Jean-Pierre Eckmann, and Elisha Moses. 2006. Hierarchical structures induce long-range dynamical correlations in written texts. Proceedings of the National Academy of Sciences, 103(21):7956– 7961. find
  12. Aryaman Arora, Clara Meister, and Ryan Cotterell. 2022. Estimating the entropy of linguistic distributions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 175–195, Dublin, Ireland. Association for Computational Linguistics. find
  13. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics. find our card
  14. Francesco Cagnetta and Matthieu Wyart. 2024. Towards a theory of how the structure of language is acquired by deep neural networks. find
  15. Werner Ebeling and Alexander Neiman. 1995. Long-range correlations between letters and sentences in texts. Physica A, 215(3):233–241. find
  16. Werner Ebeling and Thorsten Pöschel. 1994. Entropy and Long-Range correlations in literary english. Europhysics Letters, 26(4):241–246. find
  17. Kevin Ellis, Armando Solar-Lezama, and Josh Tenenbaum. 2015. Unsupervised learning by program synthesis. In Advances in Neural Information Processing Systems 28. find
  18. Édouard Gilbert and Darrell Conklin. 2007. A probabilistic context-free grammar for melodic reduction? In Proceedings of the International Workshop on Artificial Intelligence and Music, 20th International Joint Conference on Artificial Intelligence. find
  19. Peter Grassberger. 2003. Entropy estimates from insufficient samplings. arXiv:physics/0307138v2. — [arXiv:physics/0307138](https://arxiv.org/abs/physics/0307138) find
  20. Daniel Harlow, Stephen H. Shenker, Douglas Stanford, and Leonard Susskind. 2012. Tree-like structure of eternal inflation: A solvable model. Physical Review D, 85(6):063516. find
  21. John A. Hawkins. 1990. A parsing theory of word order universals. Linguistic inquiry, 21(2):223– 261. find
  22. Harold Stanley Heaps. 1978. Information Retrieval: Computational and Theoretical Aspects. Academic Press, Inc. find
  23. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  24. Daniel G. Horvitz and Donovan J. Thompson. 1952. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663– 685. find
  25. Elliot Howard-Spink, Misato Hayashi, Tetsuro Matsuzawa, Daniel Schofield, Thibaud Gruber, and Dora Biro. 2024. Nonadjacent dependencies and sequential structure of chimpanzee action during a natural tool-use task. PeerJ, 12:e18484. find
  26. Filip Jelinek, John D. Lafferty, and Robert L. Mercer. 1992. Basic methods of probabilistic context free grammars. In Speech Recognition and Understanding. find
  27. Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. 2006. Adaptor grammars: A framework for specifying compositional nonparametric bayesian models. In Advances in Neural Information Processing Systems 19. find
  28. Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy. Association for Computational Linguistics. find our card
  29. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics. find our card
  30. Bjarne Knudsen and Jotun Hein. 1999. RNA secondary structure prediction using stochastic context-free grammars and evolutionary history. Bioinformatics, 15(6):446–454. find
  31. Wentian Li. 1989a. Mutual information functions of natural language texts. Technical Report 89-10-008, Santa Fe Institute. Accessed 2026-03-02. find
  32. Wentian Li. 1989b. Spatial 1/f spectra in open dynamical systems. Europhysics Letters, 10(5):395. find
  33. Wentian Li. 1990. Mutual information functions versus correlation functions. Journal of statistical physics, 60:823–837. find
  34. Wentian Li. 1991. Expansion-modification systems: a model for spatial 1/f spectra. Physical Review A, 43(10):5240. find
  35. Henry W. Lin and Max Tegmark. 2017. Critical behavior in physics and probabilistic formal languages. Entropy, 19(7):299. find
  36. William G. Madow. 1948. On the limiting distributions of estimates based on samples from finite universes. The Annals of Mathematical Statistics, pages 535–545. find
  37. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. find our card
  38. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  39. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
  40. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations. find our card
  41. Nikolay Mikhaylovskiy and Ilya Churilov. 2023. Autocorrelations decay in texts and applicability limits of language models. arXiv:2305.06615v1. — [arXiv:2305.06615](https://arxiv.org/abs/2305.06615) find
  42. George Miller. 1955. Note on the bias of information estimates. Information theory in psychology: Problems and methods. find
  43. Kai Nakaishi and Koji Hukushima. 2024. Statistical properties of probabilistic context-sensitive grammars. Physical Review Research, 6(3):033216. find
  44. Kai Nakaishi, Yoshihiko Nishikawa, and Koji Hukushima. 2024. Critical phase transition in large language models. arXiv:2406.05335v2. — [arXiv:2406.05335](https://arxiv.org/abs/2406.05335) find
  45. National Institute for Japanese Language and Linguistics. 2016. Ninjal parsed corpus of modern japanese. Version 1.0. Available at https://npcmj.ninjal.ac.jp/interfaces/ (accessed 2025-10-29). Licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons. org/lice… ↗
  46. Ilya Nemenman, Fariel Shafee, and William Bialek. 2001. Entropy and inference, revisited. Advances in Neural Information Processing Systems 14. find
  47. Matthew Newville, Till Stensitzki, Daniel B. Allen, and Antonino Ingargiola. 2014. Lmfit: Non-linear least-square minimization and curve-fitting for python. find
  48. Timothy J. O’Donnell, Joshua B. Tenenbaum, and Noah D. Goodman. 2009. Fragment grammars: Exploring computation and reuse in language. Technical Report MIT-CSAIL-TR-2009-013, Massachusetts Institute of Technology. Accessed 2026-03-02. find
  49. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. 2014. Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. find our card
  50. Tim Sainburg and Timothy Q. Gentner. 2021. Toward a computational neuroethology of vocal communication: from bioacoustics to neurophysiology, emerging tools and future directions. Frontiers in Behavioral Neuroscience, 15:811737. find
  51. Tim Sainburg, Anna Mai, and Timothy Q. Gentner. 2022. Long-range sequential dependencies precede complex syntactic production in language acquisition. Proceedings of the Royal Society B, 289(1970):20212657. find
  52. Tim Sainburg, Brad Theilman, Marvin Thielk, and Timothy Q. Gentner. 2019. Parallels in the sequential organization of birdsong and human speech. Nature Communications, 10(1):3636. find
  53. Gideon Schwarz. 1978. Estimating the dimension of a model. The annals of statistics, pages 461– 464. find
  54. Huitao Shen. 2019. Mutual information scaling and expressive power of sequence models. arXiv:1905.04271v1. — [arXiv:1905.04271](https://arxiv.org/abs/1905.04271) find
  55. Harry E. Stanley. 1971. Introduction to Phase Transitions and Critical Phenomena. Oxford University Press. find
  56. Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2017. Do neural nets learn statistical laws behind natural language? PloS one, 12(12):e0189326. find
  57. Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2019. Evaluating computational language models with scaling properties of natural language. Computational Linguistics, 45(3):481–513. find
  58. Kumiko Tanaka-Ishii and Armin Bunde. 2016. Long-range memory in literary texts: On the universal clustering of the rare words. PLoS One, 11(11):e0164658. find
  59. Pablo Tano, Sergio Romano, Mariano Sigman, Alejo Salles, and Santiago Figueira. 2020. Towards a more flexible language of thought: Bayesian grammar updates after each concept exposure. Physical Review E, 101(4):042128. find
  60. David H. Wolpert and David R. Wolf. 1995. Estimating functions of probability distributions from a finite set of samples. Physical Review E, 52(6):6841. find
  61. Peter Worth and Susan Stepney. 2005. Growing music: Musical interpretations of l-systems. In Applications of Evolutionary Computing, pages 545–550. find
  62. Peter Worth and Susan Stepney. 2021. Recursive bayesian networks: Generalising and unifying probabilistic context-free grammars and dynamic bayesian networks. In Advances in Neural Information Processing Systems 34, pages 4370–4383. find
  63. Mason Youngblood. 2024. Language-like efficiency and structure in house finch song. Proceedings of the Royal Society B, 291(2020):20240250. find
  64. WikiText and the seven categories for NPCMJ are shown in Tables 2 and 3, respectively. find
  65. Samuel Zahl. 1977. Jackknifing an index of diversity. Ecology, 58(4):907–913. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nakaishi-2026-rethinking-relationship-between-power-law/