Elamur Research⌕
☰
home / papers / 2002 / Parsing the Wall Street Journal using a Lexical-Functional Grammar and Discriminative Estimation Techniques

Parsing the Wall Street Journal using a Lexical-Functional Grammar and Discriminative Estimation Techniques

Summary

The paper scales a stochastic parser built on a hand-written, linguistically detailed grammar up to the UPenn Wall Street Journal (WSJ) treebank. Earlier systems of this kind had two problems. Treebank annotation was too shallow to train them directly, and hand-coded grammars failed on much of free text. The authors combine the ParGram English Lexical-Functional Grammar (LFG), the XLE constraint-based parser and a conditional exponential (log-linear) disambiguation model.

Coverage reaches 100% through two fallbacks. A FRAGMENT grammar breaks unparsable input into as few well-formed chunks as possible, and a SKIMMING mode bounds the work spent on difficult sentences.

For training, the parser is run on input constrained by the treebank's POS tags and labeled brackets. The LFG analyses that agree with these brackets then serve as partially labeled data for discriminative estimation.

Evaluation counts matched predicate-argument relations rather than labeled brackets:

  • WSJ section 23: about 79% F-score against hand-annotated f-structures.
  • Brown corpus: 76% F-score against Carroll et al.'s dependency gold standard, slightly above Carroll et al.'s own parser.

The work shows that deep grammars can be trained from ordinary treebanks.

Key points

  • Grammar and lexicon: 314 rules compiled into finite-state machines with 8,759 states and 19,695 arcs. The verb lexicon holds 9,652 stems and 23,525 subcategorization-frame entries, and two guessers handle unknown words.

  • Coverage on WSJ section 23: 100% of sentences receive an analysis. 74.7% get full parses and 25.3% get FRAGMENT and/or SKIMMED parses. 7.2% of sentences were skimmed.

  • WSJ results (700 random sentences from section 23, LFG measure):

    BoundF-score
    Upper bound84.1
    Model78.6
    Lower bound75.5

    This is a 36% error reduction. Under the DR measure, the error reduction is 35%.

  • Brown results (500 sentences):

    • Dependency-only (DO) F-score is 76.1, versus 75.1 for Carroll et al. (1999).
    • The labeled DR measure gives 74.0.
    • A plain maximum-likelihood model was used here because no heldout data were available.

  • Model features and cost of fallbacks: the model uses about 1,000 property functions covering c-structure, f-structure and lexical information. Lexical features come from a clustering model of head-argument-relation tuples. Non-full parses cost roughly 4% of overall F-score.

Interesting ideas

  • I1. Shallow annotation as a filter. Instead of converting a treebank into the target formalism, the authors use its brackets and labels to filter the rich grammar's candidate parses. The surviving set serves as a soft gold standard. This approach could reuse coarse annotations, such as UD trees or chunk labels, to train richer analyzers.
  • I2. Fewest-chunk fallback. When no full parse exists, the input is covered with the smallest number of well-formed chunks, and leftover words become TOKEN chunks. This guarantees some usable output, which suits noisy or low-resource text.
  • I3. Bounded-effort processing. Once a time or memory threshold is crossed, the parser limits the work done on unfinished subtrees. Processing therefore finishes in polynomial time without dropping the sentence, a practical safeguard for any expensive search-based pipeline.
  • I4. Error reduction between bounds. A constraining grammar already pushes even a random choice of parse to a high score. The authors therefore measure the model by how much of the gap between the random lower bound and the oracle upper bound it closes. This is a fairer way to judge rerankers and selectors in general.
  • I5. Breaking scores down by parse quality. Reporting scores separately for full, fragment and skimmed parses shows how much accuracy each robustness fallback costs. This kind of diagnostic is useful for any system with graceful-degradation modes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Treebanks

Cited by

117 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 102 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1073083.1073129

@inproceedings{riezler2002parsing,
  title = {{Parsing the Wall Street Journal using a Lexical-Functional Grammar and Discriminative Estimation Techniques}},
  author = {Riezler, Stefan and King, Tracy H. and Kaplan, Ronald M. and Crouch, Richard and III, John T. Maxwell and Johnson, Mark},
  booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics},
  year = {2002},
  doi = {10.3115/1073083.1073129},
  url = {https://doi.org/10.3115/1073083.1073129}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gosse Bouma, Gertjan von Noord, and Robert Malouf. 2000. Alpino: Wide-coverage computational analysis of Dutch. In Proceedings of Computational Linguistics in the Netherlands, Amsterdam, Netherlands. find
  2. Miriam Butt, Tracy King, Maria-Eugenia Niño, and Frédérique Segond. 1999. A Grammar Writer’s Cookbook. Number 95 in CSLI Lecture Notes. CSLI Publications, Stanford, CA. find
  3. John Carroll, Guido Minnen, and Ted Briscoe. 1999. Corpus annotation for parser evaluation. In Proceedings of the EACL workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway. find our card
  4. Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. In Advances in Neural Information Processing Systems 14(NIPS’01), Vancouver. find
  5. Michael Collins. 2000. Discriminative reranking for natural language processing. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML’00), Stanford, CA. find
  6. Richard Crouch, Ronald M. Kaplan, Tracy H. King, and Stefan Riezler. 2002. A comparison of evaluation metrics for a broad-coverage stochastic parser. In Proceedings of the ”Beyond PARSEVAL” Workshop at the 3rd International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Spain… find
  7. Dan Gildea. 2001. Corpus variation and parser performance. In Proceedings of 2001 Conference on fragments 20.4 76.7 72.4 71.0 fragments 20.0 65.4 61.5 59.1 skimmed 1.4 70.3 66.6 63.0 skimmed 2.0 56.0 52.8 51.2 skimmed+fragments 3.4 61.3 56.2 55.9 skimmed+fragments 1.6 53.5 50.0 48.9 find our card
  8. Empirical Methods in Natural Language Processing (EMNLP), Pittsburgh, PA. find
  9. Tony Jebara and Alex Pentland. 1998. Maximum conditional likelihood via bound maximization and the CEM algorithm. In Advances in Neural Information Processing Systems 11 (NIPS’98). find
  10. Mark Johnson, Stuart Geman, Stephen Canon, Zhiyi Chi, and Stefan Riezler. 1999. Estimators for stochastic “unification-based” grammars. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL’99), College Park, MD. find
  11. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn treebank: Annotating predicate argument structure. In ARPA Human Language Technology Workshop. find our card
  12. John Maxwell and Ron Kaplan. 1993. The interface between phrasal and functional constraints. Computational Linguistics, 19(4):571–589. find
  13. Thomas Minka. 2001. Algorithms for maximum-likelihood logistic regression. Department of Statistics, Carnegie Mellon University. find
  14. Fernando Pereira and Yves Schabes. 1992. Inside-outside reestimation from partially bracketed corpora. In Proceedings of the 30th Annual Meeting of the Association for Computational Linguistics (ACL’92), Newark, Delaware. find our card
  15. Stefan Riezler, Detlef Prescher, Jonas Kuhn, and Mark Johnson. 2000. Lexicalized Stochastic Modeling of Constraint-Based Grammars using Log-Linear Measures and EM Training. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL’00), Hong Kong. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riezler-2002-parsing-wall-street-journal-lexical/