Elamur Research⌕
☰
home / papers / 2023 / A Cross-Linguistic Pressure for Uniform Information Density in Word Order

A Cross-Linguistic Pressure for Uniform Information Density in Word Order

Paper · 2023 · Thomas Hikaru Clark, Clara Meister, Tiago Pimentel et al.

Summary

Languages differ in word order but share statistical tendencies, which are often explained by functional pressures such as dependency length minimization. This paper asks whether uniform information density (UID) also shapes word order. UID is the idea that communication works best when information is spread evenly, without sharp peaks. The authors took dependency-parsed sentences from 10 typologically diverse languages, using UD relations with function words treated as heads. They rearranged each sentence under several counterfactual grammars, building on the procedure of Hahn et al. (2020). Transformer language models trained on the real and counterfactual corpora supplied per-word surprisal, from which three uniformity scores were derived. In SVO languages, attested orders were consistently more even than their reversed versions. Only an implausible variant that sorts words by frequency reliably beat real orders, and it does so at a large cost in expressivity. Verb-before-object grammars were generally smoother than object-before-verb ones. The findings point to a UID pressure acting through both grammar and speakers' choices, and they offer a possible account of why SVO is so widespread.

Key points

  • UID measures: Uniformity is scored three ways: within-sentence surprisal variance (UID_v), average squared word-to-word surprisal change (UID_lv), and a power mean with k > 1 (UID_p). Lower values mean greater uniformity.
  • Counterfactual grammars: The tested variants are REVERSE, EFFICIENT-VO and EFFICIENT-OV, MIN-DL-LOC and MIN-DL-OPT, RANDOM1–RANDOM5, and SORT-FREQ.
  • VO vs. OV: EFFICIENT-VO had lower surprisal variance than EFFICIENT-OV in 9 of 10 languages.
  • SOV languages: The preference for REAL over REVERSE did not hold consistently for Turkish, Hindi and Farsi. The authors attribute this to basic word order and speaker choices pulling in opposite directions.
  • Robustness checks:
    • The main corpus is Wiki40b, and a subset of languages was rerun on CC100.
    • Analyses restricted to document-initial sentences largely kept the REAL < REVERSE variance pattern.
    • Code is available at github.com/thomashikaru/word-order-uid.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

 Fields, topics and concepts

Cite

Cite as Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell and Roger Levy. 2023. A Cross-Linguistic Pressure for Uniform Information Density in Word Order. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00589

@article{clark2023cross,
  title = {{A Cross-Linguistic Pressure for Uniform Information Density in Word Order}},
  author = {Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell and Roger Levy},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00589},
  url = {https://doi.org/10.1162/tacl_a_00589}
}

References

72 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31–56. https://doi.org/10 .1177/00238309040470010201, PubMed: 15298329 find
  2. Brian Bartek, Richard L. Lewis, Shravan Vasishth, and Mason R. Smith. 2011. In search of online locality effects in sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5):1178–1198. https://doi.org/10.1037/a0024194, PubMed: 21707210 — [doi:10.1037/a0024194](https://doi.org/10.1037/a0024194) ↗
  3. Jelke Bloem. 2016. Testing the processing hypothesis of word order variation using a probabilistic language model. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity, pages 174–185, Osaka, Japan. find
  4. The COLING 2016 Organizing Committee. find
  5. Thomas Hikaru Clark, Ethan Gotlieb Wilcox, Edward Gibson, and Roger P. Levy. 2022. Evidence for availability effects on speaker choice in the Russian comparative alternation. In Proceedings of the 44th Annual Meeting of the Cognitive Science Society. find
  6. Michael Xavier Collins. 2014. Information density and dependency length as complementary cognitive models. Journal of Psycholinguistic Research, 43(5):651–681. https://doi.org/10.1007/s10936-013-9273-3, PubMed: 24077911 — [doi:10.1007/s10936-013-9273-3](https://doi.org/10.1007/s10936-013-9273-3) ↗
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f…
  8. Matthew S. Dryer. 2013. Order of subject, object and verb (v2020.3). In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. find
  9. Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2022. A measure-theoretic characterization of tight language models. arXiv preprint arXiv: 2212.10502v1. https://doi.org/10.48550/arXiv.2212.10502 ↗
  10. Nick Ellis. 2002. Frequency effects in language processing. Studies in Second Language Acquisition, 24(2):143–188. https://doi.org/10.1017/S0272263102002024 ↗
  11. August Fenk and Gertrud Fenk. 1980. Konstanz im Kurzzeitgedächtnis—Konstanz im sprachlichen Informationsfluß. Zeitschrift für ex-perimentelle und angewandte Psychologie, 27(3):400–414. find
  12. Ramon Ferrer-i-Cancho. 2004. Euclidean distance between syntactically linked words. Physical Review E, 70(5 Pt 2):056135. https://doi.org/10.1103/PhysRevE.70.056135, PubMed: 15600720 — [doi:10.1103/physreve.70.056135](https://doi.org/10.1103/physreve.70.056135) ↗
  13. Ramon Ferrer-i-Cancho, Carlos Gómez-Rodrı́guez, and Juan Luis Esteban. 2018. Are crossing dependencies really scarce? Physica A: Statistical Mechanics and its Applications, 493:311–329. https://doi.org/10.1016/j.physa.2017.10.048 ↗
  14. A. Frank and T. F. Jaeger. 2008. Speaking rationally: Uniform information density as an optimal strategy for language production. In Proceedings of the Cognitive Science Society. find
  15. Richard Futrell, Tina Hickey, Aldrin Lee, Eunice Lim, Elena Luchkina, and Edward Gibson. 2015a. Cross-linguistic gestures reflect typological universals: A subject-initial, verb-final bias in speakers of diverse languages. Cognition, 136:215–221. https://doi.org/10.1016/j .cognition.2014.11.022, Pub… — [doi:10.1016/j](https://doi.org/10.1016/j) find
  16. Richard Futrell and Roger Levy. 2017. Noisy-context surprisal as a human sentence processing cost model. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 688–698, Valencia, Spain. Association for Computationa… find
  17. Richard Futrell, Roger P. Levy, and Edward Gibson. 2020. Dependency locality as an explanatory principle for word order. Language, 96(2):371–413. https://doi.org/10.1353/lan.2020.0024 ↗
  18. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015b. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, U.S.A., 112(33):10336–10341. https://doi.org/10.1073/pnas.1502134112, PubMed: 26240370 — [doi:10.1073/pnas.1502134112](https://doi.org/10.1073/pnas.1502134112) ↗
  19. Richard Futrell, Peng Qian, Edward Gibson, Evelina Fedorenko, and Idan Blank. 2019. Syntactic dependencies correspond to word pairs with high mutual information. In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pages 3–13, Paris, France. Asso… find
  20. Georg von der Gabelentz. 1901. Die Sprachwissenschaft, ihre Aufgaben, Methoden, und bisherigen Ergebnisse, 2nd edition. C. H. Tauchnitz, Leipzig. find
  21. Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 199–206, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073117 ↗
  22. Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2018. SUD or surface-syntactic universal dependencies: An annotation scheme near-isomorphic to UD. In Proceedings of the Second Workshop on Universal Dependencies, UDW@EMNLP 2018, Brussels, Belgium, November 1, 2018, pages 66–74, Philadel…
  23. Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1–76. https://doi.org/10.1016 /S0010-0277(98)00034-1, PubMed: 9775516 find
  24. Edward Gibson. 2000. The dependency locality theory: A distance-based theory of linguistic complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium, pages 95–126, Cambridge, MA. MIT Press. find
  25. Edward Gibson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389–407. https://doi.org/10.1016/j.tics.2019.02.003, PubMed: 31006626 — [doi:10.1016/j.tics.2019.02.003](https://doi.org/10.1016/j.tics.2019.02.003) ↗
  26. Edward Gibson, Steven T. Piantadosi, Kimberly Brink, Leon Bergen, Eunice Lim, and Rebecca Saxe. 2013. A noisy-channel account of cross-linguistic word-order variation. Psychological Science, 24(7):1079–1088. https://doi.org/10.1177/0956797612463705, PubMed: 23649563 — [doi:10.1177/0956797612463705](https://doi.org/10.1177/0956797612463705) ↗
  27. Daniel Gildea and T. Florian Jaeger. 2015. Human languages order information efficiently. arXiv preprint arXiv:1510.02823. https://doi.org/10.48550/arXiv.1510.02823 — [doi:10.48550/arxiv.1510.02823](https://doi.org/10.48550/arxiv.1510.02823) ↗
  28. Daniel Gildea and David Temperley. 2007. Optimizing grammars for minimum dependency length. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 184–191, Prague, Czech Republic. Association for Computational Linguistics. different languages represent event… find
  29. Joseph H. Greenberg. 1963. Some Universals of Grammar with Particular Reference to the Order of Meaningful Elements. MIT Press, Cambridge, MA. find
  30. Daniel Grodner and Edward Gibson. 2005. Consequences of the serial nature of linguistic input for sentential complexity. Cognitive Science, 29(2):261–290. https://doi.org/10.1207/s15516709cog0000 7, PubMed: 21702774 ↗
  31. Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. 2020. Wiki-40b: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2440–2452, Marseille, France. European Language Resources Association. find
  32. Michael Hahn, Dan Jurafsky, and Richard Futrell. 2020. Universals of word order reflect optimization of grammars for efficient communication. Proceedings of the National Academy of Sciences, U.S.A., 117(5):2347–2353. https://doi.org/10.1073/pnas.1910923117, PubMed: 31964811 — [doi:10.1073/pnas.1910923117](https://doi.org/10.1073/pnas.1910923117) ↗
  33. Michael Hahn and Yang Xu. 2022. Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality. Proceedings of the National Academy of Sciences, U.S.A., 119(24):e2122604119. https://doi.org/10.1073/pnas.2122604119, PubMed: 35675428 — [doi:10.1073/pnas.2122604119](https://doi.org/10.1073/pnas.2122604119) ↗
  34. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics, pages 1–8. https://doi.org/10.3115/1073336.1073357 ↗
  35. Daniel Gildea and David Temperley. 2010. Do grammars minimize dependency length? Cognitive Science, 34(2):286–310. https://doi.org/10.1111/j.1551-6709.2009 .01073.x, PubMed: 21564213 — [doi:10.1111/j.1551-6709.2009](https://doi.org/10.1111/j.1551-6709.2009) ↗
  36. Martin Haspelmath. 2008. Parametric versus functional explanations of syntactic universals. In T. Biberauer, editor, The Limits of Syntactic Variation, pages 75–107. John Benjamins, Amsterdam. https://doi.org/10.1075/la.132.04has Susan Goldin-Meadow, Wing Chee So, Asli Özyürek, and Carolyn Myland… ↗
  37. John A. Hawkins. 1994. A Performance Theory of Order and Constituency. Cambridge University Press, Cambridge. find
  38. John A. Hawkins. 2004. Efficiency and Complexity in Grammars. Oxford University Press, Oxford. https://doi.org/10.1093/acprof :oso/9780199252695.001.0001 — [doi:10.1093/acprof](https://doi.org/10.1093/acprof) ↗
  39. John A. Hawkins. 2014. Cross-linguistic Variation and Efficiency. Oxford University Press, Oxford. https://doi.org/10.1093/acprof :oso/9780199664993.001.0001 — [doi:10.1093/acprof](https://doi.org/10.1093/acprof) ↗
  40. Jacob L. Hoover, Morgan Sonderegger, Steven T. Piantadosi, and Timothy J. O’Donnell. 2022. The plausibility of sampling as an algorithmic theory of sentence processing. PsyArXiv preprint PsyArXiv:qjnpv. https://doi.org/10.31234/osf.io/qjnpv T. Florian Jaeger. 2010. Redundancy and reduction: Speaker… — [doi:10.31234/osf.io/qjnpv](https://doi.org/10.31234/osf.io/qjnpv) ↗
  41. T. Florian Jaeger and Harry Tily. 2011. On language ‘utility’: Processing complexity and communicative efficiency. Wiley Interdisciplinary Reviews: Cognitive Science, 2(3):323–335. https://doi.org/10.1002/wcs.126, PubMed: 26302080 — [doi:10.1002/wcs.126](https://doi.org/10.1002/wcs.126) ↗
  42. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980 ↗
  43. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1318–1326, Portland, Oregon, USA. Association for Computational Linguistics.
  44. Marco Kuhlmann. 2010. Projective Dependency Structures, Lecture Notes in Computer Science, Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14568-1 3 — [doi:10.1007/978-3-642-14568-1](https://doi.org/10.1007/978-3-642-14568-1) ↗
  45. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. https://doi.org/10.1016/j.cognition .2007.05.006, PubMed: 17662975 — [doi:10.1016/j.cognition](https://doi.org/10.1016/j.cognition) ↗
  46. Roger Levy. 2018. Communicative efficiency, uniform information density, and the rational speech act theory. In Proceedings for the 40th Annual Meeting of the Cognitive Science Society, pages 684–689. https://doi.org/10 .31234/osf.io/4cgxh Roger Levy and T. Florian Jaeger. 2006. Speakers optimize in… find
  47. Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. Advances in Neural Information Processing Systems, 19:849–856. find
  48. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2):159–191. https://doi.org/10.17791/jcs.2008 .9.2.159 — [doi:10.17791/jcs.2008](https://doi.org/10.17791/jcs.2008) ↗
  49. Steven G. Luke and Kiel Christianson. 2016. Limits on lexical prediction during reading. Cognitive Psychology, 88:22–60. https://doi.org/10.1016/j.cogpsych.2016 .06.002 — [doi:10.1016/j.cogpsych.2016](https://doi.org/10.1016/j.cogpsych.2016) ↗
  50. Kyle Mahowald, Evgeniia Diachek, Edward Gibson, Evelina Fedorenko, and Richard Futrell. 2022. Experimentally measuring the redundancy of grammatical cues in transitive clauses. arXiv preprint arXiv:2201.12911. https://doi.org/10.48550/arXiv.2201 .12911 — [doi:10.48550/arxiv.2201](https://doi.org/10.48550/arxiv.2201) ↗
  51. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. https://doi.org/10.1162/coli_a_00402 ↗
  52. Luke Maurits, Dan Navarro, and Amy Perfors. 2010. Why are some word orders more common than others? A Uniform Information Density account. In Advances in Neural Information Processing Systems, pages 1585–1593. find
  53. Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisiting the Uniform Information Density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963–980, Online and Punta Cana, Dominican Republic…
  54. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021 .emnlp-main.74 — [doi:10.18653/v1/2021](https://doi.org/10.18653/v1/2021) ↗
  55. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  56. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, U.S.A., 108(9):3526–3529. https://doi.org/10.1073/pnas.1012551108, PubMed: 21278332 ↗
  57. Tiago Pimentel, Ryan Cotterell, and Brian Roark. 2021a. Disambiguatory signals are stronger in word-initial positions. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 31–41, Online. Association for Computational Lingu… find
  58. Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021b. A surprisal–duration trade-off across and within the world’s languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 949–962, Online and Punt… find
  59. Tiago Pimentel, Irene Nikkarinen, Kyle Mahowald, Ryan Cotterell, and Damián Blasi. 2021c. How (non-)optimal is the lexicon? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4426–4438, Online. Ass… find
  60. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon…
  61. Jan Rijkhoff. 1986. Word order universals revisited: The principle of head proximity. Belgian Journal of Linguistics, 1:95–125. https://doi.org/10.1075/bjl.1.05rij — [doi:10.1075/bjl.1.05rij](https://doi.org/10.1075/bjl.1.05rij) ↗
  62. Jan Rijkhoff. 1990. Explaining word order in the noun phrase. Linguistics, 28(1):5–42. https://doi.org/10.1515/ling.1990.28.1.5 — [doi:10.1515/ling.1990.28.1.5](https://doi.org/10.1515/ling.1990.28.1.5) ↗
  63. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  64. Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger P. Levy. 2022. Large-scale evidence for logarithmic effects of word predictability on reading time. PsyArXiv preprint PsyArXiv:4hyna. https://doi.org/10.31234/osf.io/4hyna Claude E. Shannon. 1948. A mathematical theory of communic… — [doi:10.31234/osf.io/4hyna](https://doi.org/10.31234/osf.io/4hyna) ↗
  65. Milan Straka and Jana Straková. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88–99, Vancouver, Canada. Association for Computational Linguistics. https://d…
  66. David Temperley and Daniel Gildea. 2018. Minimizing syntactic dependency lengths: Typological/cognitive universal? Annual Review of Linguistics, 4:1–15. https://doi.org/10 .1146/annurev-linguistics-011817 find
  67. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  68. Jennifer C. White and Ryan Cotterell. 2021. Examining the inductive bias of neural language models with artificial languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  69. Himanshu Yadav, Samar Husain, and Richard Futrell. 2021. Do dependency lengths explain constraints on crossing dependencies? Linguistics Vanguard, 7(s3). https://doi.org/10 .1515/lingvan-2019-0070 find
  70. Meilin Zhan and Roger Levy. 2018. Comparing theories of speaker choice using a model of classifier production in Mandarin Chinese. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)… find
  71. George Kingsley Zipf. 1935. The Psycho-biology of Language: An Introduction to Dynamic Philology. Houghton-Mifflin, Boston. find
  72. Ran Zmigrod, Tim Vieira, and Ryan Cotterell. 2020. Please mind the root: Decoding arborescences for dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4809–4819, Online. Association for Computational Linguistics. https://doi.o… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2023-cross-linguistic-pressure-uniform-information/