Elamur Research
☰
home / papers / 2008 / Towards Robust Semantic Role Labeling

Towards Robust Semantic Role Labeling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J08-2006.pdf

Summary

Semantic role labeling (SRL) systems are usually trained and tested on the same corpus, which can hide how poorly they generalize. The authors describe ASSERT, their SVM-based SRL system, and use it to study what happens when a labeler trained on PropBank-annotated Wall Street Journal (WSJ) text is applied to PropBanked Brown Corpus material from a different genre. ASSERT labels constituents of a syntactic parse with one-vs-all SVM classifiers over structural and lexical features. It then rescores the candidates by combining them with an argument-sequence trigram model in a Viterbi search. Accuracy falls sharply on Brown. Splitting the task into finding argument nodes and assigning their roles shows that identification, like parsing, carries over fairly well, while classification causes most of the loss. The gap remains with gold Treebank parses and when the system is trained on Brown itself. This points to Brown being more varied and polysemous, and to classification relying on lexical cues, such as predicate–headword pairs, that fit WSJ too closely. The study shows that in-domain scores overstate robustness. It argues for more diverse training data and more abstract features.

Key points

  • Data: PropBank (~300k words, WSJ portion of the Penn Treebank). Sections 02–21 are used for training (~90,000 predicates, ~250,000 arguments), Section 00 for development, and Section 23 for testing (~5,000 predicates, ~12,000 arguments). The PropBanked Brown Corpus serves as the cross-genre test set.
  • ASSERT uses TinySVM and YamCha with a degree-2 polynomial kernel and C = 1. Its SVM scores are converted to probabilities with Platt's sigmoid method. It combines these with a backed-off trigram model over argument sequences and forbids overlapping non-NULL labels.
  • Features include predicate and lemma, path, phrase type, position, voice, subcategorization, predicate clusters, and the syntactic frame (Xue and Palmer 2004). Path-like structural features matter most for identification, while lexical/semantic features dominate classification.
  • Parser errors explain little of the WSJ→Brown drop. Gold word-sense information for Brown predicates helps some verbs strongly but improves the overall test set only slightly.
  • Recommended remedies are less homogeneous training corpora and less specific features, such as broader named-entity classes and abstractions over headwords and predicates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Topics: Semantic role labeling

Cited by

43 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 42 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.2008.34.2.289

@article{pradhan2008robust,
  title = {{Towards Robust Semantic Role Labeling}},
  author = {Pradhan, Sameer S. and Ward, Wayne and Martin, James H.},
  journal = {Computational Linguistics},
  year = {2008},
  doi = {10.1162/coli.2008.34.2.289},
  url = {https://doi.org/10.1162/coli.2008.34.2.289}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bacchiani, Michiel, Michael Riley, Brian Roark, and Richard Sproat. 2006. MAP adaptation of stochastic grammars. Computer Speech and Language, 20(1):41–68. find
  2. Bikel, Daniel M., Richard Schwartz, and Ralph M. Weischedel. 1999. An algorithm that learns what’s in a name. Machine Learning, 34:211–231. find
  3. Burges, Christopher J. C. 1998. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):121–167. find
  4. Carreras, Xavier and Lluı́s Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL), pages 89–97, Boston, MA. find
  5. Carreras, Xavier and Lluı́s Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL), pages 152–164, Ann Arbor, MI. find our card
  6. Charniak, Eugene and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), Ann Arbor, MI. find our card
  7. Gildea, Daniel and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  8. Hacioglu, Kadri, Sameer Pradhan, Wayne Ward, James Martin, and Daniel Jurafsky. 2004. Semantic role labeling by tagging syntactic chunks. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL), Boston, MA. find
  9. Harabagiu, Sanda, Cosmin Adrian Bejan, and Paul Morarescu. 2005. Shallow semantics for relation extraction. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI), pages 1061–1067, Edinburgh, Scotland. find
  10. Hofmann, Thomas and Jan Puzicha. 1998. Statistical models for co-occurrence data. Memo, Massachusetts Institute of Technology Artificial Intelligence Laboratory, Cambridge, MA. find
  11. Jiang, Zheng Ping, Jia Li, and Hwee Tou Ng. 2005. Semantic argument classification exploiting argument interdependence. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI), pages 1067–1072, Edinburgh, Scotland. find
  12. Joachims, Thorsten. 1998. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the European Conference on Machine Learning (ECML), pages 137–142, Chemnitz, Germany. find our card
  13. Koomen, Peter, Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2005. Generalized inference with multiple semantic role labeling systems. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL), pages 181–184, Ann Arbor, MI. find
  14. Kučera, Henry and W. Nelson Francis. 1967. Computational Analysis of Present-day American English. Brown University Press, Providence, RI. find
  15. Kudo, Taku and Yuji Matsumoto. 2000. Use of support vector learning for chunk identification. In Proceedings of the Fourth Conference on Computational Natural Language Learning (CoNLL), pages 142–144, Lisbon, Portugal. find
  16. Kudo, Taku and Yuji Matsumoto. 2001. Chunking with support vector machines. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), Pittsburgh, PA. find
  17. Lin, Dekang. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the Seventeenth International Conference on Computational Linguistics and Thirty Sixth Annual Meeting of the Association of Computational Linguistics (COLING/ACL), pages 768–774, Montreal, Canada. find
  18. Lodhi, Huma, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Chris Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2(Feb):419–444. find
  19. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19(2):313–330. find our card
  20. Màrquez, Lluı́s, Mihai Surdeanu, Pere Comas, and Jordi Turmo. 2005. A robust combination strategy for semantic role labeling. In Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 644–651, Vancouver, British Columbia. find
  21. McClosky, David, Eugene Charniak, and Mark Johnson. 2006a. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association of Computational Linguistics (HLT/NAACL), pages 152–159, New York, NY. find our card
  22. McClosky, David, Eugene Charniak, and Mark Johnson. 2006b. Reranking and self-training for parser adaptation. In Proceedings of the Twenty First International Conference on Computational Linguistics and Forty Fourth Annual Meeting of the Association for Computational Linguistics (COLING/ACL), pages 337–344, Sydney, Australia. find our card
  23. Moschitti, Alessandro. 2006. Syntactic kernels for natural language learning: The semantic role labeling case. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association of Computational Linguistics (HLT/NAACL), pages 97–100, New York, NY. find
  24. Musillo, Gabriele and Paola Merlo. 2006. Accurate parsing of the proposition bank. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association of Computational Linguistics (HLT/NAACL), pages 101–104, New York, NY. find
  25. Narayanan, Srini and Sanda Harabagiu. 2004. Question answering based on semantic structures. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 693–701, Geneva, Switzerland. find
  26. Palmer, Martha, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  27. Platt, John. 2000. Probabilities for support vector machines. In A. Smola, P. Bartlett, B. Scholkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers. MIT Press, Cambridge, MA, pages 61–74. find
  28. Pradhan, Sameer, Kadri Hacioglu, Valerie Krugler, Wayne Ward, James Martin, and Dan Jurafsky. 2005. Support vector learning for semantic argument classification. Machine Learning Journal, 60(1):11–39. find
  29. Pradhan, Sameer, Wayne Ward, Kadri Hacioglu, James Martin, and Dan Jurafsky. 2004. Shallow semantic parsing using support vector machines. In Proceedings of the Human Language Technology Conference/ North American Chapter of the Association of Computational Linguistics (HLT/NAACL), pages 233–240, Boston, MA. find
  30. Pradhan, Sameer, Wayne Ward, Kadri Hacioglu, James Martin, and Dan Jurafsky. 2005. Semantic role labeling using different syntactic views. In Proceedings of the Forty-Third Annual Meeting of the Association for Computational Linguistics (ACL), pages 581–588, Ann Arbor, MI. find
  31. Punyakanok, Vasin, Dan Roth, Wen tau Yih, and Dav Zimak. 2005. Learning and inference over constrained output. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI), pages 1117–1123, Edinburgh, Scotland. find
  32. Surdeanu, Mihai, Sanda Harabagiu, John Williams, and Paul Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of the Forty-First Annual Meeting of the Association for Computational Linguistics (ACL), pages 8–15, Sapporo, Japan. find
  33. Toutanova, Kristina, Aria Haghighi, and Christopher Manning. 2005. Joint learning improves semantic role labeling. In Proceedings of the Forty-Third Annual Meeting of the Association for Computational Linguistics (ACL), pages 589–596, Ann Arbor, MI. find
  34. Vapnik, Vladimir. 1998. Statistical Learning Theory. Wiley, New York. find
  35. Xue, Nianwen and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 88–94, Barcelona, Spain. find
  36. Yi, Szu-ting and Martha Palmer. 2005. The integration of syntactic parsing and semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL), pages 237–240, Ann Arbor, MI. find