Elamur Research
☰
home / papers / 2001 / A Probabilistic Earley Parser as a Psycholinguistic Model

A Probabilistic Earley Parser as a Psycholinguistic Model

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N01-1021.pdf

Summary

Hale tests whether processing difficulty in human sentence comprehension can come straight from a probabilistic grammar, without any mechanism that belongs only to the processor. The paper measures cognitive load as surprisal: the share of probability mass for structural analyses that each new word rules out. Stolcke's (1995) probabilistic Earley parser over a PCFG provides prefix probabilities. The log-ratio between consecutive prefix probabilities then predicts reading time for each word. The model is fully parallel: it keeps every analysis at once, so it explains difficulty without reanalysis. Rule probabilities come from frequency counts in the parsed Brown corpus of the Penn Treebank. With both small hand-picked grammars and a large grammar taken entirely from the corpus, the parser predicts the garden-path effect in reduced relative clauses. It also predicts that object relatives are harder than subject relatives. The work matters because it shows that frequencies of phrase-level structures can stand in for dedicated processing principles. This founded the surprisal theory of sentence processing.

Key points

  • The linking hypothesis says reading time at a word reflects the log-ratio of the previous prefix probability to the current one, which is the word's surprisal.
  • The model follows three principles: strong competence (the parser uses grammar rules directly), effects of frequency, and eager processing.
  • In the garden-path sentence "the banker told about the buy-back resigned", the full corpus-derived grammar has 532 NP rules and 120 S rules. Surprisal at "resigned" is still almost 10 times higher than at the nouns.
  • From Treebank Brown counts, 4768 of 5489 relative clauses are subject relatives (≈0.87) and 721 are object relatives (≈0.13).
  • Using a GPSG-style grammar for relative clauses, mean surprisal is about 5.0 for object relatives and about 2.1 for subject relatives.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

341 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 334 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hale2001probabilistic,
  title = {{A Probabilistic Earley Parser as a Psycholinguistic Model}},
  author = {Hale, John},
  booktitle = {Second Meeting of the North American Chapter of the Association for Computational Linguistics},
  year = {2001},
  url = {https://aclanthology.org/N01-1021/}
}

References

34 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fred Attneave. 1959. Applications of Information Theory to Psychology: A summary of basic concepts, methods and results. Holt, Rinehart and Winston. find
  2. Thomas G. Bever. 1970. The cognitive basis for linguistic structures. In J.R. Hayes, editor, Cognition and the Development of Language, pages 279–362. Wiley, New York. find
  3. Taylor L. Booth and Richard A. Thompson. 1973. Applying probability measures to abstract languages. IEEE Transactions on Computers, C-22(5). find
  4. Joan Bresnan. 1982. Introduction: Grammars as mental representations of language. In Joan Bresnan, editor, The Mental Representation of Grammatical Relations, pages xvii,lii. MIT Press, Cambridge, MA. find
  5. Eugene Charniak. 1993. Statistical Language Learning. MIT Press. find
  6. Ciprian Chelba and Frederick Jelinek. 1998. Exploiting syntactic structure for language modelling. In Proceedings of COLING-ACL ’98, pages 225–231, Montreal. find
  7. Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113–124. find
  8. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press, Cambridge MA. find
  9. Jay Earley. 1970. An efficient context-free parsing algorithm. Communications of the Association for Computing Machinery, 13(2), February. find
  10. Janet Dean Fodor and Fernanda Ferreira, editors. 1998. Reanalysis in sentence processing, volume 21 of Studies in Theoretical Psycholingustics. Kluwer, Dordrecht. find
  11. Marilyn Ford. 1989. Parsing complexity and a theory of parsing. In Greg N. Carlson and Michael K. Tanenhaus, editors, Linguistic Structure in Language Processing, pages 239–272. Kluwer. find
  12. Gerald Gazdar, Ewan Klein, Geoffrey Pullum, and Ivan Sag. 1985. Generalized Phrase Structure Grammar. Harvard University Press, Cambridge, MA. find
  13. Edward Gibson and Neal J. Pearlmutter. 1998. find
  14. Constraints on sentence processing. Trends in Cognitive Sciences, 2:262–268. Edward Gibson and Carson Schütze. 1999. Disambiguation preferences in noun phrase conjunction do not mirror corpus frequency. Journal of Memory and Language. find
  15. Edward Gibson. 1998. Linguistic complexity: locality of syntactic dependencies. Cognition, 68:1–76. find
  16. Ulf Grenander. 1967. Syntax-controlled probabilities. Technical report, Brown University Division of Applied Mathematics, Providence, RI. find
  17. Frederick Jelinek and John D. Lafferty. 1991. Computation of the probability of initial substring generation by stochastic context-free grammars. Computational Linguistics, 17(3). find
  18. Maryellen C. MacDonald, Neal J. Pearlmutter, and Mark S. Seidenberg. 1994. Lexical nature of syntactic ambiguity resolution. Psychological Review, 101(4):676–703. find
  19. James McClelland and Mark St. John. 1989. Sentence comprehension: A PDP approach. Language and Cognitive Processes, 4:287–336. find
  20. Don C. Mitchell, Fernando Cuetos, Martin M.B. Corley, and Marc Brysbaert. 1995. Exposure-based models of human parsing: Evidence for the use of coarse-grained (nonlexical) statistical records. Journal of Psycholinguistic Research, 24(6):469–488. find
  21. Srini Narayanan and Daniel Jurafsky. 1998. Bayesian models of human sentence processing. In Proceedings of the 19th Annual Conference of the Cognitive Science Society, University of Wisconsin-Madson. find
  22. Mark-Jan Nederhof, Anoop Sarkar, and Giorgio Satta. 1998. Prefix probabilities from stochastic tree adjoining grammars. In Proceedings of COLING-ACL ’98, pages 953–959, Montreal. find
  23. Brian Roark and Mark Johnson. 1999. Broad coverage predictive parsing. Presented at the 12th Annual CUNY Conference on Human Sentence Processing, March. find
  24. Stuart Shieber and Mark Johnson. 1993. Variations on incremental interpretation. Journal of Psycholinguistic Research, 22(2):287–318. find
  25. Stuart Shieber. 1985. Evidence against the context-freeness of natural language. Linguistics and Philosophy, 8:333–343. find
  26. Stephen Soule. 1974. Entropies of probabilistic grammars. Information and Control, 25(57–74). find
  27. Edward Stabler. 1991. Avoid the pedestrian’s paradox. In Robert C. Berwick, Steven P. Abney, and Carol Tenny, editors, Principle-Based Parsing: computation and psycholinguistics, Studies in Linguistics and Philosophy, pages 199–237. Kluwer, Dordrecht. find
  28. Mark Steedman. 1992. Grammars and processors. find
  29. Technical Report TR MS-CIS-92-52, University of Pennsylvania CIS Department. find
  30. Andreas Stolcke. 1995. An efficient probabilistic context-free parsing algorithm that computes prefix probabilities. Computational Linguistics, 21(2). find
  31. Andreas Stolcke. 1997. Linguistic knowledge and empirical methods in speech recognition. AI Magazine, 18(4):25–31. find
  32. Whitney Tabor, Cornell Juliano, and Michael Tanenhaus. 1997. Parsing in a dynamical system: An attractor-based account of the interaction of lexical and structural constraints in sentence processing. Language and Cognitive Processes, 12(2/3):211–271. find
  33. Eric Wanner and Michael Maratsos. 1978. An ATN approach to comprehension. In Morris Halle, Joan Bresnan, and George A. Miller, editors, Linguistic Theory and Psychological Reality, chapter 3, pages 119–161. MIT Press, Cambridge, Massachusetts. find
  34. C.S. Wetherell. 1980. Probabilistic languages: A review and some open questions. Computing Surveys, 12(4). find