Elamur Research
☰
home / papers / 2008 / The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies

The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies

What it is about

The 2008 CoNLL task joined syntactic dependency parsing and semantic role labelling in one dependency formalism. The English data convert the Penn Treebank constituents into dependencies, and the semantic layer covers both verbal predicates (PropBank) and nominal predicates (NomBank). Each system had to produce a syntactic dependency tree and the arguments of each predicate. The paper defines the subtasks, the closed and open challenges, and the evaluation measures: labelled attachment score, labelled F1 for the semantic dependencies, exact match of the whole sentence and perfect-proposition F1, which are much lower than the token scores. The main analysis is that nominal predicates are harder than verbal ones: the average gap in labelled F1 is 7.84 points on WSJ and Brown together, and 12.36 points on Brown alone, so nominal roles are more sensitive to domain change. Dividing the semantic F1 by the LAS gives a normalised score that isolates the role labeller from the parser. The paper also finds that wh-movement and split-clause links, which are non-projective, are picked out by simple patterns (a wh-word, a pair of commas).

Ideas

For Mova

The chapters are languages/en/11-nouns.md (nominal predicates, relational nouns), languages/en/13-verbs.md (verbal predicates and argument structure), languages/en/25-relative-clauses.md and languages/en/26-questions.md (wh-movement and the non-projective links). The code is en/src/parse.rs for the attachment and the non-projective rule, prag/src/roles.rs and prag/src/frames.rs for the role readers, and en parse-eval for the scores. The measures are the verb-noun gap on the school texts and the Early Modern texts, the non-projective precision and recall on UD EWT dev, and the normalised role score.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W08-2121.pdf

Summary

The CoNLL-2008 shared task brought two research lines that earlier CoNLL editions had treated separately (semantic role labeling in 2004–2005 and dependency parsing in 2006–2007) into one English-only, dependency-based task. Participating systems had to do three things: produce a labeled syntactic dependency tree, find and disambiguate predicates, and attach role-labeled arguments to each predicate. The organizers built the data by converting Penn Treebank constituents into dependencies. The conversion added richer structure, including appositions, named-entity-derived links and better long-distance relations, and it mapped every PropBank and NomBank argument, covering both verbal and nominal predicates. Systems were ranked by a macro-averaged F1 that combines syntactic LAS with semantic labeled F1. There was a closed challenge restricted to the supplied data and an open challenge that allowed external resources. Twenty-two teams took part. The task matters because it set a common, reproducible benchmark for joint syntactic–semantic analysis. Its predicate-to-argument-head representation also suits applications such as information extraction and retrieval, where the full argument span is often unnecessary.

Key points

  • Data format: CoNLL-style columns with at least 11 fields per token.
    • Hyphenated and slashed words are split into sub-tokens, because NomBank annotates parts of words. The original unsplit forms are kept in separate columns.
    • Predicted POS tags come from SVMTool, and the tags on the training set were produced with 5-fold cross-validation.
    • Lemmas come from the WordNet lemmatizer.
  • Open-track extras:
    • CoNLL-2003 and BBN named-entity tags, with the BBN tags produced by 2-fold cross-validation because of overlap with the Penn Treebank.
    • WordNet supersenses.
    • MaltParser heads and labels.
  • Official scoring:
    • Syntax is scored with LAS.
    • Semantics is scored with labeled precision, recall and F1 over predicate→argument dependencies, plus a predicate→ROOT link that carries the predicate sense.
    • The overall score is the harmonic mean of macro precision and macro recall, with W_sem = 0.5.
  • Unofficial measures:
    • Exact Match, the share of fully correct sentences.
    • Perfect Proposition F1.
    • The ratio of semantic labeled F1 to LAS.
  • Participating systems: Johansson and Che, two of the three highest-ranked entries, each built distinct models for verbal and for nominal predicates. Che et al. also used separate inference constraints for each model.

Interesting ideas

  • Encode predicate senses as edges to a virtual ROOT. This turns sense disambiguation into one more labeled dependency, so a wrong sense costs one edge instead of the whole proposition. The trick is useful wherever a structured output mixes classification and attachment decisions.
  • Normalize a downstream score by its upstream score. Dividing semantic F1 by LAS separates the quality of the role labeler from the parser it depends on. The same idea applies to any pipeline evaluation, such as NER on top of tokenization or relation extraction on top of parsing.
  • Pair tolerant metrics with strict ones. Reporting Exact Match and Perfect Proposition F1 next to token-level scores rewards systems that get whole structures right. This makes the benefit of joint learning visible, which averaged per-token scores can hide.
  • Predict features on training data via cross-validation. Generating automatic features, such as POS or NE tags, through k-fold cross-validation gives the training data the same noise level as test-time inputs. This is a simple safeguard against train/test mismatch in any stacked system.
  • Keep both split and unsplit token layers. Storing the original and sub-word tokenizations side by side lets annotation work below the word level without losing the original text. This is relevant for morphologically rich languages such as Ukrainian.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1596324.1596352

@inproceedings{surdeanu2008conll,
  title = {{The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies}},
  author = {Surdeanu, Mihai and Johansson, Richard and Meyers, Adam and Màrquez, Lluís and Nivre, Joakim},
  booktitle = {CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning},
  year = {2008},
  doi = {10.3115/1596324.1596352},
  url = {https://doi.org/10.3115/1596324.1596352}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. X. Carreras. 2007. Experiments with a Higher-Order Projective Dependency Parser. In Proc. of CoNLL-2007 Shared Task. http://code.google.com/p/whatswrong/ ↗
  2. X. Carreras and L. Màrquez. 2005. Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling. In Proc. of CoNLL-2005. find our card
  3. X. Carreras and L. Màrquez. 2004. Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling. In Proc. of CoNLL-2004. find
  4. W. Che, Z. Li, Y. Hu, Y. Li, B. Qin, T. Liu and S. Li. 2008. A Cascaded Syntactic and Semantic Dependency Parsing System. In Proc. of CoNLL-2008 Shared Task. find
  5. E. Chen, L. Shi and D. Hu. 2008. Probabilistic Model for Syntactic and Semantic Dependency Parsing. In Proc. of CoNLL-2008 Shared Task. find
  6. Chinchor, N. and P. Robinson. 1998. MUC-7 Named Entity Task Definition. In Proc. of Seventh Message Understanding Conference (MUC-7). http://www.itl.nist.gov/iaui/894.02/related projects/ /muc/proceedings/muc 7 toc.html. ↗
  7. Chomsky, Noam. 1981. Lectures on Government and Binding. Foris Publications, Dordrecht. find
  8. Y.J. Chu and T.H. Liu. 1965. On the Shortest Arborescence of a Directed Graph. In Science Sinica, 14:1396-1400. find
  9. M. Ciaramita, G. Attardi, F. Dell’Orletta, and M. Surdeanu. 2008. DeSRL: A Linear-Time Semantic Role Labeling System. In Proc. of CoNLL-2008 Shared Task. find
  10. M. Ciaramita and Y. Altun. 2006. Broad Coverage Sense Disambiguation and Information Extraction with a Supersense Sequence Tagger. In Proc. of EMNLP. find
  11. M. Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find our card
  12. J. Edmonds. 1967. Optimum Branchings. In Journal of Research of the National Bureau of Standards, 71B:233–240. find
  13. J. Eisner. 2000. Bilexical Grammars and Their Cubic-Time Parsing Algorithms. New Developments in Parsing Algorithms, Kluwer Academic Publishers. find
  14. W. N. Francis and H. Kuĉera. 1964. Brown Corpus. Manual of Information to accompany A Standard Corpus of Present-Day Edited American English, for use with Digital Computers. Revised 1971, Revised and Amplified 1979. find
  15. C. Fellbaum, editor. 1998. WordNet: An electronic lexical database. MIT Press. find
  16. J. Giménez and L. Màrquez. 2004. SVMTool: A general POS tagger generator based on Support Vector Machines. In Proc. of LREC. find
  17. K. Hacioglu. 2004. Semantic Role Labeling Using Dependency Trees. In Proc. of COLING-2004. find
  18. J. Henderson, P. Merlo, G. Musillo and I. Titov. 2008. A Latent Variable Model of Synchronous Parsing for Syntactic and Semantic Dependencies. In Proc. of CoNLL-2008 Shared Task. find
  19. R. Johansson and P. Nugues. 2008. Dependency-based Syntactic–Semantic Analysis with PropBank and NomBank. In Proc. of CoNLL-2008 Shared Task. find
  20. R. Johansson and P. Nugues. 2007. Extended Constituent-to-Dependency Conversion for English. In Proc. of NODALIDA. find our card
  21. X. Lluı́s and L. Màrquez. 2008. A Joint Model for Parsing Syntactic and Semantic Dependencies. In Proc. of CoNLL-2008 Shared Task. find
  22. D. Magerman. 1994. Natural Language Parsing as Statistical Pattern Recognition. Ph.D. thesis, Stanford University. find
  23. M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: the Penn Treebank. Computational Linguistics, 19. find our card
  24. R. McDonald, F. Pereira, K. Ribarov and J. Hajic. 2005. Non-Projective Dependency Parsing using Spanning Tree Algorithms In Proc. of HLT-EMNLP. find our card
  25. A. Meyers, R. Grishman, M. Kosaka, and S. Zhao. 2001. Covering Treebanks with GLARF. In Proc. of the ACL/EACL 2001 Workshop on Sharing Tools and Resources for Research and Education. find
  26. Meyers, A., R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The NomBank Project: An Interim Report. In NAACL/HLT 2004 Workshop Frontiers in Corpus Annotation, Boston. find
  27. J. Nivre, J. Hall, J. Nilsson and G. Eryigit. 2006. Labeled Pseudo-Projective Dependency Parsing with Support Vector Machines. In Proc. of CoNLL-X Shared Task. find
  28. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, D. Yuret. 2007. find
  29. The CoNLL 2007 Shared Task on Dependency Parsing. In Proc. of CoNLL-2007. find
  30. J. Nivre, J. Hall, J. Nilsson, A. Chanev, G. Eryiğit, S. Kübler, S. Marinov, and E. Marsi. 2007b. Malt-Parser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135. find
  31. M. Palmer, D. Gildea, and P. Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics, 31(1). find our card
  32. S. Riedel and I. Meza-Ruiz. 2008. Collective Semantic Role Labelling with Markov Logic. In Proc. of CoNLL-2008 Shared Task. find
  33. Y. Samuelsson, O. Täckström, S. Velupillai, J. Eklund, M. Fishel and M. Saers. 2008. Mixing and Blending Syntactic and Semantic Dependencies. In Proc. of CoNLL-2008 Shared Task. find
  34. W. Sun, H. Li and Z. Sui. 2008. The Integration of Dependency Relation Classification and Semantic Role Labeling Using Bilayer Maximum Entropy Markov Models. In Proc. of CoNLL-2008 Shared Task. find
  35. E. F. Tjong Kim San and F. De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In Proc. of CoNLL-2003. find
  36. D. Vickrey and D. Koller. 2008. Applying Sentence Simplification to the CoNLL-2008 Shared Task. In Proc. of CoNLL-2008 Shared Task. find
  37. R. Weischedel and A. Brunstein. 2005. BBN pronoun coreference and entity type corpus. Technical report, Linguistic Data Consortium. find
  38. H. Yamada and Y. Matsumoto. 2003. Statistical Dependency Analysis with Support Vector Machines. In Proc. of IWPT. find our card
  39. Y. Zhang, R. Wang and H. Uszkoreit. 2008. Hybrid Learning of Dependency Structures from Heterogeneous Linguistic Resources. In Proc. of CoNLL-2008 Shared Task. find
  40. Zhao, S. 2005. Information Extraction from Multiple Syntactic Sources. Ph.D. thesis, NYU. find