Elamur Research
☰
home / papers / 2003 / Accurate Unlexicalized Parsing

Accurate Unlexicalized Parsing

What it is about

The paper asks how far a PCFG with no lexical items can go on the Penn Treebank. A vanilla treebank grammar parses badly because its independence assumptions are too strong. Adding simple linguistic state splits, such as parent annotation, marking of base noun phrases, tag splits for function words and horizontal markovization, breaks those assumptions, and each split is kept only if it improves F1 on held-out data. The final unlexicalized grammar scores 86.3 F1 on section 23 for sentences of up to 40 words, which is above the early lexicalized PCFGs and close to the state of the art, and it is smaller, faster to parse and easier to replicate. The paper shows that lexical head statistics are not the only source of accuracy.

Ideas

For Mova

The splits in I1 target the words where the tagger is most often unsure, so the chapters are 10-word-classes and 12-pronouns-determiners-numerals, with 07-punctuation-capitals-numbers for the punctuation tags. The code is en/src/ptag.rs with the TnT prior in en/src/tag.rs, and en/src/parse.rs for idea I2. Measure first: the lexical ablation of idea I3, as it tells whether tag splits or features are the cheaper gain.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P03-1054.pdf

Summary

The paper challenges the common belief that strong probabilistic constituency parsing requires lexicalized PCFGs, where phrasal nodes carry head words. Klein and Manning argue that much of the benefit attributed to word-level statistics actually comes from structural information. They test this by enriching a plain treebank PCFG with hand-chosen, linguistically motivated state splits. Training uses sections 2–21 of the WSJ Penn treebank, a small development set from section 22, and section 23 for testing. The method combines vertical and horizontal markovization (parent and ancestor annotation plus head-outward binarization) with targeted category refinements. Examples include separating finite from non-finite VPs and marking base NPs and nodes that contain verbs. All rule probabilities are plain unsmoothed maximum-likelihood estimates.

The final grammar reaches roughly 86.3 F1 (LP/LR) on sentences of up to 40 words. That score surpasses two early lexicalized systems, Collins (1996) and Magerman (1995), but still falls short of Charniak (1997) and Collins (1999). The result sets a much higher baseline for unlexicalized models. Such grammars are also smaller, easier to interpret and reproduce, and faster to parse.

Key points

  • Data split:
    • Training: WSJ Penn treebank sections 2–21.
    • Development: the first 20 files of section 22 (393 sentences), used in a partly manual hill-climb over features.
    • Test: all of section 23.
  • Baseline and markovization: The raw treebank grammar scores 72.62 F1 on the dev set. In this view, the standard grammar is simply the setting v = 1 (no ancestor context) and h = ∞ (rules never split). Adding ancestors and limiting sibling history raises scores into the high 70s.
  • Largest single gain: SPLIT-VP annotates each VP with its head tag, with finite forms merged into VBF. This lifted cumulative dev F1 to 85.72, a gain of 2.66 points.
  • Attachment-oriented features:
    • Base-NP marking without Collins' extra unary node took cumulative dev F1 to 86.04. Collins' original version actually lowered the baseline by 0.37.
    • DOMINATES-V raised it to 86.91.
    • RIGHT-REC-NP gave the final dev F1 of 87.04.
  • Test results and efficiency:
    • Section 23, sentences of 40 words or fewer: LP 86.9 and LR 85.7. The results section reports F1 as 86.32, while the abstract gives 86.36.
    • Sentences of up to 100 words: F1 85.7.
    • A simple Java CKY parser handled all of section 23 exhaustively within 1 GB of memory, taking about 3 seconds per average-length sentence.
    • Unlexicalized parsing is O(n³), compared with O(n⁵) for naive lexicalized parsing.

Interesting ideas

  • Context as two adjustable settings: Parent annotation (adding context) and rule markovization (removing context) are treated as two dials on one scale, vertical history v and horizontal history h. Framing a model's independence assumptions as tunable parameters gives a systematic search space instead of ad hoc tweaks. This could help any structured model, for example taggers or grammars for Ukrainian treebanks.
  • Linguistic error analysis as feature discovery: Each split was motivated by an observed failure, such as present-tense verbs wrongly taking bare infinitive complements or PPs attached too high. Pairing error inspection with small, interpretable features is a cheap and transparent alternative to adding model complexity.
  • Indirect proxies for hard properties: Encoding linear distance in a PCFG would explode the number of states. The authors instead used cheap indicators that correlate with attachment height, such as "contains a verb" or "is a base NP." When a property is hard to model directly, a correlated structural flag may capture most of its value.
  • Building a strong simple baseline first: The authors argue that the field overestimated the value of lexicalization because no well-tuned unlexicalized baseline existed. Investing in a strong, simple baseline before claiming that a complex component helps is a good evaluation habit for any project.
  • Smoothing unknown words with word classes: Unknown-word tag probabilities were backed off to classes built from capitalization, suffixes, digits, and other character features. The authors show this choice alone shifts all reported scores (72.62 vs. 72.09 on the raw grammar). The trick carries over well to morphologically rich languages, and the observation is a reminder to report preprocessing effects.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1075096.1075150

@inproceedings{klein2003accurate,
  title = {{Accurate Unlexicalized Parsing}},
  author = {Klein, Dan and Manning, Christopher D.},
  booktitle = {Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics},
  year = {2003},
  doi = {10.3115/1075096.1075150},
  url = {https://doi.org/10.3115/1075096.1075150}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James K. Baker. 1979. Trainable grammars for speech recognition. In D. H. Klatt and J. J. Wolf, editors, Speech Communication Papers for the 97th Meeting of the Acoustical Society of America, pages 547–550. find
  2. Taylor L. Booth and Richard A. Thomson. 1973. Applying probability measures to abstract languages. IEEE Transactions on Computers, C-22:442–450. find
  3. Sharon A. Caraballo and Eugene Charniak. 1998. New figures of merit for best-first probabilistic chart parsing. Computational Linguistics, 24:275–298. find
  4. Eugene Charniak, Sharon Goldwater, and Mark Johnson. 1998. Edge-based best-first chart parsing. In Proceedings of the Sixth Workshop on Very Large Corpora, pages 127–133. find
  5. Eugene Charniak. 1996. Tree-bank grammars. In Proc. of the 13th National Conference on Artificial Intelligence, pp. 1031–1036. find
  6. Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In Proceedings of the 14th National Conference on Artificial Intelligence, pp. 598–603. find
  7. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In NAACL 1, pages 132–139. find
  8. Eugene Charniak. 2001. Immediate-head parsing for language models. In ACL 39. find
  9. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT Press, Cambridge, MA. find
  10. Michael John Collins. 1996. A new statistical parser based on bigram lexical dependencies. In ACL 34, pages 184–191. find
  11. M. Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, Univ. of Pennsylvania. find
  12. Jason Eisner and Giorgio Satta. 1999. Efficient parsing for bilexical context-free grammars and head-automaton grammars. In ACL 37, pages 457–464. find
  13. Marilyn Ford, Joan Bresnan, and Ronald M. Kaplan. 1982. A competence-based theory of syntactic closure. In Joan Bresnan, editor, The Mental Representation of Grammatical Relations, pages 727–796. MIT Press, Cambridge, MA. find
  14. Daniel Gildea. 2001. Corpus variation and parser performance. In 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  15. Donald Hindle and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103–120. find
  16. Mark Johnson. 1998. PCFG models of linguistic tree representations. Computational Linguistics, 24:613–632. find our card
  17. Dan Klein and Christopher D. Manning. 2001. Parsing with treebank grammars: Empirical bounds, theoretical models, and the structure of the Penn treebank. In ACL 39/EACL 10. find
  18. David M. Magerman. 1995. Statistical decision-tree models for parsing. In ACL 33, pages 276–283. find
  19. Andrew Radford. 1988. Transformational Grammar. Cambridge University Press, Cambridge. find
  20. Dana Ron, Yoram Singer, and Naftali Tishby. 1994. The power of amnesia. Advances in Neural Information Processing Systems, volume 6, pages 176–183. Morgan Kaufmann. find