Elamur Research
☰
home / papers / 2005 / Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling

Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling

What it is about

The overview of the 2005 shared task on SRL over PropBank. Systems label the arguments of given target verbs with roles. The training data is the Wall Street Journal sections 02 to 21 with full syntactic parses from two parsers, and the test is section 23 plus three Brown sections (ck01 to 03) to test robustness across corpora. Nineteen systems took part, all in the closed setting. The best system reached F1 79.44 on the WSJ test. Every system lost about 10 F1 points on the Brown set, which the authors attribute to parser error propagation in the pipeline. The top four systems were combinations of several systems working on different parses, and the best single system reached 75.04 on the combined test. A seven-rule baseline (for example, the noun phrase before the verb is A0, the one after is A1) is far below all systems. The argument is strict: a span is right only if its boundaries and its label are both right, and the verb itself is not scored.

Ideas

For Mova

The chapters are languages/en/21-clause-structure.md and languages/en/13-verbs.md. The code is prag/src/roles.rs and tools/treeagree. The measure is span-and-label F1 on school sentences, run twice to check determinism, and the cross-genre gate in I1.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W05-0620.pdf

Summary

This paper describes the CoNLL-2005 competition, whose shared task was devoted to Semantic Role Labeling (SRL). For each target verb in an English sentence, a system must find the phrases that fill its semantic roles and label them with PropBank roles. The 2004 edition gave systems only partial syntax. This edition added three things: full parse trees from two alternative parsers, a much larger training set from the WSJ part of the Penn TreeBank annotated with PropBank, and an extra out-of-domain test set from the Brown corpus.

Systems are scored by precision, recall and F1 over whole arguments. An argument counts only if both its span and its label are correct.

Nineteen teams took part, all in the closed challenge. The strongest system, Punyakanok et al. (2005), reached F1 79.44 on the WSJ test set, roughly 10 points above the best 2004 result. On Brown, every system lost about 10 F1 points. The authors present this as evidence that errors pile up through pipelined NLP architectures when the domain changes.

Key points

  • Data splits: training uses WSJ sections 02-21, development uses section 24, and the WSJ test uses section 23. Brown sections ck01-03 provide the cross-corpus test. The training portion has 39,832 sentences, 950,028 tokens, 90,750 propositions and 239,858 arguments.
  • Input annotations: words, PoS tags, base chunks, clauses, named entities, and full syntactic trees from two parsers. Role labels come from PropBank-1.0. They cover numbered arguments (A0–A5, AA), 13 adjunct types (AM-), references (R-) and the verb (V).
  • Results:
    • The best individual system (Surdeanu and Turmo, 2005) got F1 75.04 on the combined WSJ+Brown test, about 3 points below the best combined system.
    • With the same architecture, Màrquez et al. (2005) scored 73.57 on development using partial parsing, versus 75.75 using full parsing.
  • Effect of each test set: most systems scored 1–2 F1 points higher on the WSJ test than on development. On Brown they dropped about 10 points, even though the Brown baseline was higher.
  • Scoring tool: the official scorer is srl-eval.pl. Data, software and system outputs were published on the task website.

Interesting ideas

  • Drop trivial items from the metric. The verb argument is excluded from scoring because it is almost always given by the target verb, and counting it would inflate the scores. Any benchmark can apply this: remove near-free items so the metric reflects the hard part of the task.
  • Separate closed and open tracks. A closed track lets only the training data and the organizers' preprocessors be used, while an open track allows external resources. The closed track makes systems directly comparable, and the open track measures what extra knowledge adds.
  • Test on a second domain. Adding an out-of-domain test set (Brown next to WSJ) exposed a roughly 10-point drop that in-domain testing would have hidden. Testing on a different source is a cheap way to check robustness, for example for low-resource languages with mixed-genre data.
  • Combine systems built on different parses. The top systems merged several SRL outputs, each built on different syntactic analyses. This makes up for any single parse being wrong and improves both coverage and robustness.
  • Give everyone the same predicted inputs. The organizers supplied the outputs of the syntactic preprocessors, including several alternative parsers. This makes it possible to measure how much each layer of syntax helps, and it keeps comparisons fair.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1706543.1706571

@inproceedings{carreras2005introduction,
  title = {{Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling}},
  author = {Carreras, Xavier and Màrquez, Lluís},
  booktitle = {Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005)},
  year = {2005},
  doi = {10.3115/1706543.1706571},
  url = {https://doi.org/10.3115/1706543.1706571}
}

References

36 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xavier Carreras and Lluı́s Màrquez. 2003. Phrase recognition by filtering and ranking with perceptrons. In Proceedings of RANLP-2003, Borovets, Bulgaria. find
  2. Xavier Carreras and Lluı́s Màrquez. 2004. Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling. In Proceedings of CoNLL-2004. find
  3. Eugene Charniak. 2000. A maximum-entropy inspired parser. In Proceedings of NAACL-2000. find
  4. Wanxiang Che, Ting Liu, Sheng Li, Yuxuan Hu, and Huaijun Liu. 2005. Semantic role labeling system using maximum entropy classifier. In Proceedings of CoNLL-2005. find
  5. Hai Leong Chieu and Hwee Tou Ng. 2003. Named entity recognition with a maximum entropy approach. In Proceedings of CoNLL-2003, Edmonton, Canada. find
  6. Trevor Cohn and Philip Blunsom. 2005. Semantic role labelling with tree conditional random fields. In Proceedings of CoNLL-2005. find
  7. Michael Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find our card
  8. Charles J. Fillmore, Charles Wooters, and Collin F. Baker. 2001. Building a large lexical databank which provides deep semantics. In Proceedings of the Pacific Asian Conference on Language, Informa tion and Computation, Hong Kong, China. find
  9. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
  10. Jesús Giménez and Lluı́s Màrquez. 2003. Fast and accurate part-of-speech tagging: The svm approach revisited. In Proceedings of RANLP-2003, Borovets, Bulgaria. find
  11. Aria Haghighi, Kristina Toutanova, and Christopher Manning. 2005. A joint model for semantic role labeling. In Proceedings of CoNLL-2005. find
  12. Richard Johansson and Pierre Nugues. 2005. Sparse bayesian classification of predicate arguments. In Proceedings of CoNLL-2005. find
  13. Chi-San Lin and Tony C. Smith. 2005. Semantic role labeling via consensus in pattern-matching. In Proceedings of CoNLL-2005. find
  14. Ken Litkowski. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Proceedings of the Senseval-3 ACL-SIGLEX Workshop. find our card
  15. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19. find our card
  16. Lluı́s Màrquez, Pere Comas, Jesús Giménez, and Neus Català. 2005. Semantic role labeling as sequential tagging. In Proceedings of CoNLL-2005. find
  17. Tomohiro Mitsumori, Masaki Murata, Yasushi Fukuda, Kouichi Doi, and Hirohumi Doi. 2005. Semantic role labeling using support vector machines. In Proceedings of CoNLL-2005. find
  18. Alessandro Moschitti, Ana-Maria Giuglea, Bonaventura Coppola, and Roberto Basili. 2005. Hierarchical semantic role labeling. In Proceedings of CoNLL-2005. find
  19. Alessandro Moschitti. 2004. A study on convolution kernel for shallow semantic parsing. In Proceedings of the 42nd Annual Conference of the Association for Computational Linguistics (ACL-2004). find
  20. Necati Ercan Ozgencil and Nancy McCracken. 2005. Semantic role labeling using libSVM. In Proceedings of CoNLL-2005. find
  21. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1). find our card
  22. Kyung-Mi Park and Hae-Chang Rim. 2005. Maximum entropy based semantic role labeling. In Proceedings of CoNLL-2005. find
  23. Simone Paolo Ponzetto and Michael Strube. 2005. Semantic role labeling using lexical statistical information. In Proceedings of CoNLL-2005. find
  24. Sameer Pradhan, Kadri Hacioglu, Valerie Krugler, Wayne Ward, James Martin, and Daniel Jurafsky. 2005a. Support vector learning for semantic argument classification. Machine Learning. Special issue on Speech and Natural Language Processing. To appear. find
  25. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H. Martin, and Daniel Jurafsky. 2005b. Semantic role chunking combining complementary syntactic views. In Proceedings of CoNLL-2005. find
  26. Vasin Punyakanok, Dan Roth, Wen-Tau Yih, and Dav Zimak. 2004. Semantic role labeling via integer linear programming inference. In Proceedings of the International Conference on Computational Linguistics (COLING). find
  27. Vasin Punyakanok, Peter Koomen, Dan Roth, and Wen tau Yih. 2005. Generalized inference with multiple semantic role labeling systems. In Proceedings of CoNLL-2005. find
  28. Mihai Surdeanu and Jordi Turmo. 2005. Semantic role labeling using complete syntactic analysis. In Proceedings of CoNLL-2005. find
  29. Mihai Surdeanu, Sanda Harabagiu, John Williams, and Paul Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of ACL 2003, Sapporo, Japan. find
  30. Charles Sutton and Andrew McCallum. 2005. Joint parsing and semantic role labeling. In Proceedings of CoNLL-2005. find
  31. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2003. find
  32. Erik Tjong Kim Sang, Sander Canisius, Antal van den Bosch, and Toine Bogers. 2005. Applying spelling error correction techniques for improving semantic role labelling. In Proceedings of CoNLL-2005. find
  33. Tzong-Han Tsai, Chia-Wei Wu, Yu-Chun Lin, and Wen-Lian Hsu. 2005. Exploiting full parsing information to label semantic roles using an ensemble of me and svm via integer linear programming. In Proceedings of CoNLL-2005. find
  34. Sriram Venkatapathy, Akshar Bharati, and Prashanth Reddy. 2005. Inferring semantic roles using subcategorization frames and maximum entropy model. In Proceedings of CoNLL-2005. find
  35. Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  36. Szu-ting Yi and Martha Palmer. 2005. The integration of syntactic parsing and semantic role labeling. In Proceedings of CoNLL-2005. find