Elamur Research⌕
☰
home / papers / 2026 / A Syntactic and Semantic Probe into Language Evolution based on Large Language Models

A Syntactic and Semantic Probe into Language Evolution based on Large Language Models

Paper · 2026 · Hao Pang, Changcheng Li and Yingxue Liu

Summary

Studies testing whether languages evolve toward lower communicative effort have mostly measured dependency distance on hand-annotated treebanks. Such treebanks are scarce for historical texts and low-resource languages, and this approach looks at syntax alone. The authors replace parsing with signals read directly from pretrained language models.

Attention-based Structural Distance (ASD) uses final-layer self-attention. It merges heads with weights that favour low-entropy, sharply focused heads, discards special tokens, and measures how far in linear position each word's attention reaches. Averaging over sentences gives a document-level score, MASD. Semantic Space Distance (SSD) adds a semantic dimension by measuring how far apart part-of-speech groups lie in embedding space.

Across several diachronic, multilingual corpora totalling about 70 million tokens, ASD falls over time while SSD rises. The authors interpret this as languages becoming more structurally compact and functionally more distinguishable, in line with Zipf-style efficiency. They argue that LLMs and humans share processing tendencies, so annotation-free model-based probes can support research on language change.

Key points

  • ASD aggregates final-layer attention heads using softmax weights over each head's negative mean entropy. It then combines the attention matrix with an |i−j| positional distance matrix to compute per-token, per-sentence and document-level (MASD) scores.
  • The data total roughly 70 million tokens from multiple diachronic and multilingual corpora. Reported analyses include the State of the Union Address with BERT and RSC data with Qwen3-Embedding-0.6B.
  • Main finding: ASD decreases and SSD increases over time, and the two are negatively associated. The trends hold across different model architectures, which the authors read as a property of the data rather than a model artifact.
  • Cognitive interpretation: in a predictive processing system, clearer separation of POS categories narrows expectations about upcoming words, while shorter attention spans lower integration cost.
  • Limitations: temporal coverage and typological diversity are limited, and the link between attention patterns and specific cognitive processes remains unresolved. Planned work includes parallel diachronic cross-linguistic and cross-dialectal corpora and hierarchical attention networks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksCultural Studies · 20 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksHistorical linguistics & language change · 249 works

Cite

Cite as Hao Pang, Changcheng Li and Yingxue Liu. 2026. A Syntactic and Semantic Probe into Language Evolution based on Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2095

@inproceedings{pang2026syntactic,
  title = {{A Syntactic and Semantic Probe into Language Evolution based on Large Language Models}},
  author = {Hao Pang, Changcheng Li and Yingxue Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2095},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2095}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computationa… find
  2. Michael Dambacher, Martin Rolfs, Karsten Göllner, Reinhold Kliegl, and Arthur M. Jacobs. 2009. Event-related potentials reveal rapid verification of predicted visual input. PLoS ONE, 4(3):e5047. find
  3. Katherine A. DeLong, Thomas P. Urbach, and Marta Kutas. 2005. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience, 8(8):1117– 1121. find
  4. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Suzanne Dikker and Liina Pylkkänen. 2013. Predicting language: MEG evidence for lexical preactivation. Brain and Language, 127(1):55–64. find
  7. Suzanne Dikker, Hugh Rabagliati, Thomas A. Farmer, and Liina Pylkkänen. 2010. Early occipital sensitivity to syntactic category is based on form typicality. Psychological Science, 21(5):629–634. find
  8. Kara D. Federmeier. 2007. Thinking ahead: The role and roots of prediction in language comprehension. Psychophysiology, 44(4):491–505. find
  9. Stefan Fischer, Jörg Knappen, Katrin Menzel, and Elke Teich. 2020. The Royal Society Corpus 6.0: Providing 300+ years of scientific writing for humanistic study. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 794–802, Marseille, France. European Language Resources Ass… find
  10. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336–10341. find
  11. Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68(1):1–76. find
  12. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. CoRR, abs/1901.05287.
  13. Peizhu Gong, Jin Liu, Yurong Xie, Minjie Liu, and Xiliang Zhang. 2023. Enhancing context representations with part-of-speech information and neighboring signals for question classification. Complex Intell. Syst., 9:6191–6209. find
  14. Peter Hagoort and Peter Indefrey. 2014. The neurobiology of language beyond single words. Annual Review of Neuroscience, 37:347–362. find
  15. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
  16. W. Liu, M. Xiang, and N. Ding. 2025. Active use of latent tree-structured sentence representation in humans and large language models. Nature Human Behaviour. find
  17. Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in bert track syntactic dependencies? ArXiv, abs/1911.12246. find
  18. Akira Miyake and Priti Shah, editors. 1999. Models of Working Memory: Mechanisms of Active Maintenance and Executive Control. Cambridge University Press, Cambridge. find
  19. Falk Huettig. 2015. Four central questions about prediction in language processing. Brain Research, 1626:118–135. Predictive and Attentive Processing in Perception and Action. find
  20. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. find
  21. T. Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 61(1):23–62. find
  22. Sabine Schulte im Walde and Diego Frassinelli. 2022. Distributional measures of semantic abstraction. Frontiers in Artificial Intelligence, Volume 4 - 2021. find
  23. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–437…
  24. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Lear…
  25. Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. 2016. Redefining part-of-speech classes with distributional semantic models. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 115–125, Berlin, Germany. Association for Computational Linguistics. find
  26. Sara Laszlo and Kara D. Federmeier. 2009. A beautiful day in the neighborhood: An event-related potential study of lexical relationships and prediction in context. Journal of Memory and Language, 61(3):326– 338. find
  27. Ellen F. Lau, Kirsten Weber, Alexandre Gramfort, Matti S. Hämäläinen, and Gina R. Kuperberg. 2016. Spatiotemporal signatures of lexical-semantic prediction. Cerebral Cortex, 26(4):1377–1387. find
  28. Lei Lei and Matthew L. Jockers. 2020. Normalized dependency distance: Proposing a new measure. Journal of Quantitative Linguistics, 27(1):62–79. find
  29. Bowen Li, Taeuk Kim, Reinald Kim Amplayo, and Frank Keller. 2020. Heads-up! unsupervised constituency parsing via self-attention heads. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Nat… find
  30. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. The Journal of Cognitive Science, 9:159–191. find
  31. Haitao Liu, Chunshan Xu, and Junying Liang. 2016. Dependency length minimization: Puzzles and promises. Glottometrics, 33:35–38. find
  32. Jos J. A. Van Berkum, Colin M. Brown, Pienie Zwitserlood, Vibeke Kooijman, and Peter Hagoort. 2005. Anticipating upcoming words in discourse: Evidence from ERPs and reading times. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(3):443–467. find
  33. Cyma Van Petten and Barbara J. Luka. 2012. Prediction during language comprehension: Benefits, costs, and ERP components. International Journal of Psychophysiology, 83(2):176–190. find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Cur…
  35. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
  36. C. Westbury and G. Hollis. 2019. Conceptualizing syntactic categories as semantic categories: Unifying part-of-speech identification and semantics using co-occurrence vector averaging. Behav Res, 51:1371– 1398. find
  37. Nicole Y. Y. Wicha, Eva M. Moreno, and Marta Kutas. 2004. Anticipating words and their gender: An event-related brain potential study of semantic integration, gender expectancy, and gender agreement in spanish sentence reading. Journal of Cognitive Neuroscience, 16(7):1272–1288. find
  38. Roel M. Willems, Stefan L. Frank, Annabel D. Nijhof, Peter Hagoort, and Antal van den Bosch. 2015. Prediction during natural language comprehension. Cerebral Cortex, 26(6):2506–2516. find
  39. Victor H. Yngve. 1960. A model and an hypothesis for language structure. Proceedings of the American Philosophical Society, 104(5):444–466. find
  40. George K. Zipf. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley Press, Boston. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pang-2026-syntactic-semantic-probe-into-language/