Elamur Research⌕
☰
home / papers / 2021 / Revisiting the Uniform Information Density Hypothesis

Revisiting the Uniform Information Density Hypothesis

Paper · 2021 · Clara Meister, Tiago Pimentel, Patrick Haller et al.

Summary

The uniform information density (UID) hypothesis claims that people prefer utterances in which information, measured as surprisal, is spread evenly. Most earlier evidence comes from language production. Two questions have stayed open: how evenness should be measured, and over what span it should hold (phrase, sentence, document or the whole language). The authors test UID from the comprehension side. They use sentence-level reading times from reading corpora, including eye-tracking data, and sentence acceptability judgments. They fit regression models whose predictor is the sum of surprisals raised to a power k, scanning a fine-grained range of k. They then compare competing ways of quantifying uniformity.

For reading times, a linear surprisal effect fits well, but values of k slightly above 1 fit about as well and cannot be excluded. Acceptability shows a clearer super-linear pattern: sentences with uneven information are rated worse. Measures based on a language-wide mean surprisal beat local ones, which supports the channel-capacity view of UID. The work gives a principled basis for picking UID metrics in psycholinguistics and NLP.

Key points

  • Core predictor: the predictor is the sum of surprisal^k over a sentence.

    • For acceptability, correlation rises when k > 1 (Fig. 1).
    • For reading times, k slightly above 1 matches or beats k = 1, most visibly in the Provo corpus.

  • Analysis design: the analysis works at the sentence level and counts re-reading after the first pass. A finer k grid replaces the coarser quadratic test of Smith and Levy (2013).

  • Operationalizations compared: three families were tested:

    • super-linear surprisal;
    • deviation from a target mean µ_c (global variability);
    • differences between neighbouring surprisals (local variability, after Collins 2014).

    The super-linear form explains psychometric data best.

  • Scope of the mean: the target mean was computed at sentence, document and language level, with µ_lang estimated over WikiText 103. Higher-level means predicted reading times better in every corpus.

  • Robustness check and limitations: replacing surprisal with negative raw probability (prompted by Brothers and Kuperberg 2021) gave worse fits.

    • The data cover only written English, plus Dutch in the appendix.
    • Code: github.com/rycolab/revisiting-uid.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🎓Massachusetts Institute of TechnologyCambridge · United States159 works · 65,157 citations🎓University of ZurichZurich · Switzerland50 works · 11,138 citations🎓ETH ZurichZurich · Switzerland36 works · 4,534 citations🎓University of PotsdamPotsdam · Germany36 works · 5,236 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksNeuroscience · 95 worksCognitive Neuroscience · 95 works

🧭 Topics 1Research subjects and languages the work is about.Psycholinguistics & language acquisition · 66 works

Cite

Cite as Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell and Roger Levy. 2021. Revisiting the Uniform Information Density Hypothesis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.74

@inproceedings{meister2021revisiting,
  title = {{Revisiting the Uniform Information Density Hypothesis}},
  author = {Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell and Roger Levy},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.74},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.74}
}

References

61 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31–56. find
  2. Alan Bell, Daniel Jurafsky, Eric Fosler-Lussier, Cynthia Girand, Michelle Gregory, and Daniel Gildea. 2003. Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. The Journal of the Acoustical Society of America, 113(2):1001–1024. find
  3. Jelke Bloem. 2016. Testing the processing hypothesis of word order variation using a probabilistic language model. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 174–185. find
  4. The COLING 2016 Organizing Committee. find
  5. Trevor Brothers and Gina R. Kuperberg. 2021. Word predictability effects are linear, not logarithmic: Implications for probabilistic models of sentence comprehension. Journal of Memory and Language, 116:104174. find
  6. Harm Brouwer, Hartmut Fitz, and John Hoeks. 2010. Modeling the noun phrase versus sentence coordination ambiguity in Dutch: Evidence from surprisal theory. In Proceedings of the 2010 Workshop on Cognitive Modeling and Computational Linguistics, pages 72–80. Association for Computational Linguistics. find
  7. Michael Xavier Collins. 2014. Information density and dependency length as complementary cognitive models. Journal of Psycholinguistic Research, 43(5):651–681. find
  8. Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior Research Methods, 49(2):602–615. find
  9. Christophe Coupé, Yoon Mi Oh, Dan Dediu, and François Pellegrino. 2019. Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science Advances, 5(9). find
  10. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. Association for Comput… find
  11. Wietse de Vries and Malvina Nissim. 2021. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846. Association for Computational Linguistics.
  12. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  14. August Fenk and Gertraud Fenk. 1980. Konstanz im Kurzzeitgedächtnis - Konstanz im sprachlichen Informationsfluß? Zeitschrift für Experimentelle und Angewandte Psychologie, 27(3):400–414. find
  15. Irene Fernandez Monsalve, Stefan L. Frank, and Gabriella Vigliocco. 2012. Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 398–408. Association for Computational Linguistics… find
  16. Victoria Fossum and Roger Levy. 2012. Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2012), pages 61–69. Association for Computational Linguistics. find
  17. A. F. Frank and T. Jaeger. 2008. Speaking rationally: Uniform information density as an optimal strategy for language production. In the Annual Meeting of the Cognitive Science Society. find
  18. Stefan L. Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829– 834. find
  19. Stefan L. Frank, Leun J. Otten, Giulia Galli, and Gabriella Vigliocco. 2013. Word surprisal predicts n400 amplitude during reading. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 878–883. Association for Computational Lingui… find
  20. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2018. The Natural Stories Corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Associa… find
  21. Dmitriy Genzel and Eugene Charniak. 2002. Entropy rate constancy in text. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 199–206. Association for Computational Linguistics.
  22. Adam Goodkind and Klinton Bicknell. 2018a. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18.
  23. Adam Goodkind and Klinton Bicknell. 2018b. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18. Association for Computational Linguistics.
  24. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  25. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197. Association for Computational Linguistics.
  26. T. Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 61:23–62. find
  27. Ayush Jain, Vishal Singh, Sidharth Ranjan, Rajakrishnan Rajkumar, and Sumeet Agarwal. 2018. Uniform Information Density effects on syntactic choice in Hindi. In Proceedings of the Workshop on Linguistic Complexity and Natural Language Processing, pages 38–48. Association for Computational Linguistic… find
  28. Alan Kennedy, Robin Hill, and Joel Pynte. 2003. The Dundee Corpus. In Proceedings of the 12th European Conference on Eye Movements. find
  29. Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  30. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. find
  31. Roger Levy. 2005. Probabilistic Models of Word Order and Syntactic Discontinuity. Ph.D. thesis, Stanford University. find
  32. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177. find
  33. Roger Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, volume 19. MIT Press. find
  34. Roger P. Levy. 2018. Communicative efficiency, Uniform Information Density, and the Rational Speech Act Theory. In 40th Annual Meeting of the Cognitive Science Society, pages 684–689. Cognitive Science Society. find
  35. Steven G. Luke and Kiel Christianson. 2018. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826–833. find
  36. Kyle Mahowald, Evelina Fedorenko, Steven T. Piantadosi, and Edward Gibson. 2013. Info/information theory: Speakers choose shorter words in predictive contexts. Cognition, 126(2):313–318. find
  37. Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. If beam search is the answer, what was the question? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173–2185. Association for Computational Linguistics.
  38. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations.
  39. Jeff Mitchell, Mirella Lapata, Vera Demberg, and Frank Keller. 2010. Syntactic and semantic factors in processing difficulty: An integrated measure. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 196– 206. Association for Computational Linguistics. find
  40. Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochastic language modelling. Computer Speech and Language, 8(1):1–38. find
  41. Irene Nikkarinen, Tiago Pimentel, Damián Blasi, and Ryan Cotterell. 2021. Modelling the unigram distribution. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3721– 3729. Association for Computational Linguistics. find
  42. François Pellegrino, Ioana Chitoran, Egidio Marsico, and Christophe Coupé. 2011. A cross-language perspective on speech information rate. Language, 87(3):539–558. find
  43. Dirk Pijpops, Dirk Speelman, Stefan Grondelaers, and Freek Van de Velde. 2018. Comparing explanations for the complexity principle: evidence from argument realization. Language and Cognition, 10(3):514–543. find
  44. Tiago Pimentel, Clara Meister, Elizabeth Salesky, Simone Teufel, Damián Blasi, and Ryan Cotterell. 2021. A surprisal–duration trade-off across and within the world’s languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational L… find
  45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  46. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3):372–422. find
  47. Alfréd Rényi. 1961. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 547–561. University of California Press. find
  48. Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. 2020. The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing. BioRxiv. find
  49. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. Association for Computational Linguistics.
  50. Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. find
  51. Les Sikos, Clayton Greenberg, Heiner Drenhaus, and Matthew W Crocker. 2017. Information density of encodings: The role of syntactic variation in comprehension. In 39th Annual Meeting of the Cognitive Science Society. Cognitive Science Society. find
  52. Processing, pages 5191–5202. Association for Computational Linguistics. find
  53. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 1707–1713. Cognitive Science Society. find
  54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…
  55. Meilin Zhan and Roger Levy. 2018. Comparing theories of speaker choice using a model of classifier production in Mandarin Chinese. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1997–2005. Assoc… find
  56. Meilin Zhan and Roger P. Levy. 2019. Availability-based production predicts speakers’ real-time choices of Mandarin classifiers. In 41st Annual Meeting of the Cognitive Science Society, pages 1268–1274. Cognitive Science Society. find
  57. Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302–319. find
  58. Sascha Topolinski and Fritz Strack. 2009. The architecture of intuition: fluency and affect determine intuitive judgments of semantic and visual coherence and judgments of grammaticality in artificial grammar learning. Journal of Experimental Psychology: General, 138(1):39. find
  59. Fatemeh Torabi Asr and Vera Demberg. 2015. Uniform surprisal at the level of discourse relations: Negation markers and discourse connective omission. In Proceedings of the 11th International Conference on Computational Semantics, pages 118–128. Association for Computational Linguistics. find
  60. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  61. Jason Wei, Clara Meister, and Ryan Cotterell. 2021. A cognitive regularizer for language modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meister-2021-revisiting-uniform-information-density-hypothesis/