Elamur Research⌕
☰
home / papers / 2004 / Shallow Semantic Parsing using Support Vector Machines

Shallow Semantic Parsing using Support Vector Machines

Summary

The paper addresses shallow semantic parsing, also called semantic role labeling: for each verb in a sentence, find the constituents that act as its arguments and assign them PropBank labels such as ARG0–ARG5 and ARGM adjuncts. The authors start from the Gildea and Jurafsky (2002) framework. They replace its statistical classifier with Support Vector Machines and then extend the feature set. Some new features come from earlier work (named entities, head word POS), and others are new (verb clusters, partial path, verb sense, prepositional-phrase head words). Training and testing use the WSJ-based PropBank with the standard section split, on both hand-corrected TreeBank parses and Charniak parser output. On the same features, SVMs clearly beat a decision tree and an interpolated probabilistic model. The full system improves on earlier published results for argument identification and labeling. The authors also annotated a small AQUAINT test set to measure how well the system carries over to other newswire. Performance drops substantially there, and they trace part of the drop to lower coverage of sparse lexical and path features.

Key points

  • Data: PropBank (July 2002 release), trained on WSJ Sections 02–21 and tested on Section 23.
    • Training: about 51,000 sentences and 132,000 arguments.
    • Feb 2004 release: about 85,000 training sentences and 250,000 arguments.
  • Setup: one-vs-all SVMs built with TinySVM and YamCha, using a degree-2 polynomial kernel and C=1.
  • Classifier comparison: with identical features, argument classification accuracy is 88% for the SVM, 79% for the decision tree of Surdeanu et al. (2003), and 77% for Gildea and Palmer (2002).
  • Baseline on hand-corrected parses (all arguments):
    • Identification F1: 90.4.
    • Classification accuracy: 87.9%.
    • Identification + classification F1: 80.8, rising to 81.6 after removing overlapping arguments.
  • Final system, identification + classification: F1 87 on hand-corrected parses and 79 on automatic parses.
  • AQUAINT test set (400 PropBank-annotated sentences from NYT, AP and Xinhua): identification + classification F1 falls to 63.3, even though about 99% of predicates were seen in training.

Interesting ideas

  • I1. Two-stage training to handle NULL nodes. About 80% of parse nodes are not arguments. The authors train one cheap NULL vs NON-NULL filter on all data and train the remaining one-vs-all classifiers only on the roughly 20% of nodes that pass the filter. This saves a lot of training time without hurting accuracy, and the same trick could help any heavily imbalanced candidate-classification task.
  • I2. Enforcing output constraints with calibrated scores. Raw SVM scores are converted to probabilities with Platt's sigmoid. When predicted arguments overlap, only the most confident one is kept. This is a simple, general way to apply structural constraints on top of independent local decisions.
  • I3. Generalizing sparse features. Two features trade detail for coverage:
    • "Partial path" keeps only the tree path up to the lowest common ancestor of the constituent and the predicate.
    • Verb clusters (64 classes) let the model handle predicates it has rarely or never seen. This kind of back-off is useful wherever rich features are too sparse.
  • I4. Feature coverage as a diagnostic for domain shift. The authors compare how often key feature combinations seen in training reappear in the in-domain and out-of-domain test sets. This gives a concrete, interpretable explanation for cross-domain performance loss and is easy to reproduce for other corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Semantic role labeling

Cited by

130 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pradhan2004shallow,
  title = {{Shallow Semantic Parsing using Support Vector Machines}},
  author = {Pradhan, Sameer S. and Ward, Wayne H. and Hacioglu, Kadri and Martin, James H. and Jurafsky, Dan},
  booktitle = {Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004},
  year = {2004},
  url = {https://aclanthology.org/N04-1030/}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel M. Bikel, Richard Schwartz, and Ralph M. Weischedel. 1999. An algorithm that learns what’s in a name. Machine Learning, 34:211– 231. find
  2. Don Blaheta and Eugene Charniak. 2000. Assigning function tags to parsed text. In NAACL, pages 234–240. find
  3. Eugene Chaniak. 2001. Immediate-head parsing for language models. In ACL-01. find our card
  4. John Chen and Owen Rambow. 2003. Use of deep linguistics features for the recognition and labeling of semantic arguments. EMNLP-03. find
  5. Michael John Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. find
  6. K. Daniel, Y. Schabes, M. Zaidel, and D. Egedi. 1992. A freely available wide coverage morphological analyzer for English. In COLING-92. find
  7. Michael Fleischman and Eduard Hovy. 2003. A maximum entropy approach to framenet tagging. In HLT-03. find
  8. Dan Gildea and Julia Hockenmaier. 2003. Identifying semantic roles using combinatory categorial grammar. In EMNLP-03. find
  9. Daniel Gildea and Daniel Jurafsky. 2000. Automatic labeling of semantic roles. In ACL-00, pages 512–520. find our card
  10. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  11. Daniel Gildea and Martha Palmer. 2002. The necessity of syntactic parsing for predicate argument recognition. In ACL-02. find
  12. Kadri Hacioglu and Wayne Ward. 2003. Target word detection and semantic role chunking using support vector machines. In HLT-03. find
  13. Kadri Hacioglu, Sameer Pradhan, Wayne Ward, James Martin, and Dan Jurafsky. 2003. Shallow semantic parsing using support vector machines. Technical Report TR-CSLR-2003-1, Center for Spoken Language Research, Boulder, Colorado. find
  14. Thomas Hofmann and Jan Puzicha. 1998. Statistical models for co-occurrence data. Memo, MIT AI Laboratory. find
  15. Paul Kingsbury, Martha Palmer, and Mitch Marcus. 2002. Adding semantic annotation to the Penn Treebank. In HLT-02. find
  16. Taku Kudo and Yuji Matsumoto. 2000. Use of support vector learning for chunk identification. In CoNLL-00. find
  17. Taku Kudo and Yuji Matsumoto. 2001. Chunking with support vector machines. In NAACL-01. find our card
  18. LDC. 2002. The AQUAINT Corpus of English News Text, Catalog no. LDC2002t31. find
  19. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In COLING-98. find our card
  20. David Magerman. 1994. Natural Language Parsing as Statistical Pattern Recognition. Ph.D. thesis, Stanford University, CA. find
  21. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn TreeBank: Annotating predicate argument structure. find our card
  22. John Platt. 2000. Probabilities for support vector machines. In A. Smola, P. Bartlett, B. Scolkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers. MIT press. find
  23. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James Martin, and Dan Jurafsky. 2003. Semantic role parsing: Adding semantic structure to unstructured text. In ICDM-03. find
  24. Mihai Surdeanu, Sanda Harabagiu, John Williams, and Paul Aarseth. 2003. Using predicate-argument structures for information extraction. In ACL-03. find our card
  25. Cynthia A. Thompson, Roger Levy, and Christopher D. Manning. 2003. A generative model for semantic role labeling. In ECML-03. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pradhan-2004-shallow-semantic-parsing-support-vector/