Elamur Research⌕
☰
home / papers / 2002 / The Necessity of Parsing for Predicate Argument Recognition

The Necessity of Parsing for Predicate Argument Recognition

Summary

Statistical systems that assign semantic roles, such as the FrameNet-based labeler of Gildea and Jurafsky (2002), depend on automatic parse trees. Until now, no one could measure how much parser mistakes hurt them. Propbank annotates sentences from the Wall Street Journal part of the Penn Treebank, so hand-checked trees exist for all of its data, and that makes the measurement possible. The authors port the FrameNet role labeler to a preliminary Propbank release. Its features are phrase type, tree path, position, voice and head word, and it combines them by interpolating probability distributions over a backoff lattice. They compare three kinds of input: gold Treebank trees, output of a statistical parser (Collins 1997), and a flat chunked representation. When the system must also find argument boundaries, gold trees reach 71.1% precision and 64.4% recall, and automatic parses reach 57.7/50.0. Chunks perform far worse. The study shows that full syntactic parsing genuinely helps role recognition and that progress in parsing should carry over directly to semantic interpretation.

Key points

  • The Propbank data used contains 26,138 predicate-argument structures with 65,364 arguments. Roles are defined per verb: core arguments are numbered (ARG0, ARG1…), and adjunct-like ones get labels such as temporal or locative. Word sense and roleset were not annotated in this release.
  • The model estimates P(role | phrase type, path, position, voice, head word, predicate). It does this by linearly interpolating the most specific sub-distributions that have data in a backoff lattice. Voice is detected with 10 passive patterns, each requiring a passive auxiliary plus a past participle.
  • Results with unknown argument boundaries (precision/recall):
    • gold parse: 71.1/64.4
    • auto parse: 57.7/50.0
    • chunks: 27.6/22.0
    • chunks with relaxed scoring: 49.5/35.1
  • The chunk-based system uses gold-standard chunking and labels single chunks only. Sentential complements (11% of the data) and prepositional phrases (10%) always span several chunks, so the system can never get them fully right.
  • In an error analysis, both systems miss a long-distance ARG0, which would require Treebank traces. The chunker also mislabels a nearby NP and misses a verb-attached Manner PP, which the tree path (VB↑VP↓PP) captures.

Interesting ideas

  • I1. Three-way input ablation (gold / automatic / flat): running the same downstream model on gold, predicted, and simplified input separates the cost of preprocessing errors from the value of the representation itself. This design transfers to any pipeline built on UD treebanks or other annotated parses.
  • I2. Relaxed "first chunk correct" scoring: crediting a prediction when only the argument's start is labeled correctly shows how much loss comes from boundary detection versus role choice. This is a cheap diagnostic for span-based tasks.
  • I3. Tree path as an atomic string feature: encoding the route from predicate to constituent as one symbol, such as VB↑VP↑S↓NP, packs grammatical relation and attachment into a single sparse feature. It is easy to compute from dependency or constituency trees.
  • I4. Backoff lattice interpolation: mixing the most specific feature-subset distributions that have counts handles sparse per-predicate data without a large model. This is useful for low-resource languages or rare predicates.
  • I5. Representation upper bound: counting how many gold arguments can map onto a single unit (chunk) at all gives a ceiling on performance before training anything. This helps decide early whether a flatter, cheaper representation is adequate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Semantic role labeling

Institutions

🎓University of PennsylvaniaPhiladelphia · United States143 works · 56,645 citations

Cite

Cite as Daniel Gildea and Martha Palmer. 2002. The Necessity of Parsing for Predicate Argument Recognition. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1073083.1073124

@inproceedings{gildea2002necessity,
  title = {{The Necessity of Parsing for Predicate Argument Recognition}},
  author = {Daniel Gildea and Martha Palmer},
  booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics},
  year = {2002},
  doi = {10.3115/1073083.1073124},
  url = {https://doi.org/10.3115/1073083.1073124}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hiyan Alshawi, editor. 1992. The Core Language Engine . MIT Press, Cambridge, MA. find
  2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of COLING/ACL , pages 86{90, Montreal, Canada. our card
  3. D. M. Bikel, S. Miller, R. Schwartz, and R. Weischedel. 1997. Nymble: a high-performance learning name- nder. In Proceedings of the Fifth Conference on Applied Natural Language Processing . find
  4. Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In AAAI-97 , pages 598{603, Menlo Park, August. AAAI Press. find
  5. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the 35th Annual Meeting of the ACL , pages 16{23, Madrid, Spain. our card
  6. Charles J. Fillmore. 1976. Frame semantics and the nature of language. In Annals of the New York Academy of Sciences: Conference on the Origin and Development of Language and Speech , volume 280, pages 20{32. find
  7. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics , in press. our card
  8. Jerry R. Hobbs, Douglas Appelt, John Bear, David Israel, Megumi Kameyama, Mark E. Stickel, and Mabry Tyson. 1997. FASTUS: A cascaded nite-state transducer for extracting information from natural-language text. In Emmanuel Roche and Yves Schabes, editors, Finite-State Language Processing , pages 383{… find
  9. Christopher R. Johnson, Charles J. Fillmore, Esther J. Wood, Josef Ruppenhofer, Margaret Urban, Miriam R. L. Petruk, and Collin F. Baker. 2001. The FrameNet project: Tools for lexicon building. Version 0.7, http://www.icsi.berkeley.edu/~framenet/book.html. ↗
  10. Paul Kingsbury and Martha Palmer. 2002. From Treebank to PropBank. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC-2002) , Las Palmas, Canary Islands, Spain. our card
  11. John La erty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random elds: Probabilistic models for segmenting and labeling sequence data. In Machine Learning: Proceedings of the Eighteenth International Conference (ICML 2001) , Stanford, California. find
  12. Andrew McCallum, Dayne Freitag, and Fernando Pereira. 2000. Maximum entropy Markov models for information extraction and segmentation. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML 2000) , pages 591{598, Stanford, California. find
  13. Scott Miller, Michael Crystal, Heidi Fox, Lance Ramshaw, Richard Schwartz, Rebecca Stone, Ralph Weischedel, and the Annotation Group. 1998. Algorithms that learn to extract information { BBN: Description of the SIFT system as used for MUC-7. In Proceedings of the Seventh Message Understanding Confer… find
  14. Soumya Ray and Mark Craven. 2001. Representing sentence structure in hidden markov model for information extraction. In Seventeenth International Joint Conference on Arti cial Intelligence (IJCAI-01) , Seattle, Washington. find
  15. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task: Chunking. In Proceedings of CoNLL-2000 and LLL-2000 , Lisbon, Portugal. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gildea-2002-necessity-parsing-predicate-argument-recognition/