CoNLL-X Shared Task on Multilingual Dependency Parsing

What it is about

The shared task asked for dependency parsers for 13 languages, trained and tested on the same data converted into one ten-column format (the parser predicts HEAD and DEPREL). The test set in each language was made to hold about 5,000 scoring tokens, and the score was labelled attachment, with punctuation tokens left out. Nineteen groups submitted parsed test data, and LAS ranged from 37.8% on Turkish to 91.7% on Japanese. The paper describes the conversion of each treebank, the evaluation and the approaches used, and draws conclusions about which languages, treebanks and annotation choices are hard for any dependency parser, including the role of morphology, data size and non-projectivity.

Ideas

I1. en ud-eval: check that the evaluation excludes exactly the tokens the official script excludes (FORM made only of Unicode punctuation, with a test file that includes the characters underscore and percent sign), and compare en ud-eval with the official script on EWT dev; the scores must match.
I2. en ud-eval: when comparing English treebanks, cut each test set to the same number of scoring tokens (about 5,000, as in this task) so that differences in LAS are not driven by test size.
I3. Use the per-language spread of the shared task as a stress set: report en LAS on the least and most morphologically rich languages only as context, and never as a claim about English.

For Mova

The ideas touch the scorer behind en ud-eval and how Mova reports results across treebanks. The chapter that gives the reporting context is languages/en/08-national-varieties, since treebanks differ in their annotation choices. Measure first I1, a scorer check that costs nothing and makes every later LAS number comparable.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W06-2920.pdf

Summary

Before this shared task, dependency parsers were seldom compared on identical data outside English. There was no agreed metric (labeled or unlabeled, separate root accuracy), no standard train/test split, no common procedure for converting constituency treebanks to dependencies, and each treebank used its own format. CoNLL-X addressed this by converting treebanks for 13 languages into one tab-separated, ten-column format based on Malt-TAB and encoded in UTF-8. It also fixed one official metric, labeled attachment score (LAS), computed over non-punctuation tokens. Teams had roughly two to three months to build a system that could be trained for every language, then four days to parse unseen test sets. Nineteen groups submitted results using many different parsing approaches. Scores ranged from 37.8% (Turkish) to 91.7% (Japanese). The variation was consistent enough for the organizers to investigate which properties of a language, its treebank or its annotation conventions affect how difficult it is to parse. The data, conversion software and the eval.pl scorer were released as shared resources for future multilingual parsing work.

Key points

13 languages were covered. 12 were obligatory and Bulgarian was optional. Six of the sources were native dependency treebanks, for example the Prague Dependency Treebank (PDT) for Czech. The others were converted from different annotation styles.
The data format has 10 fields: ID, FORM, LEMMA, CPOSTAG, POSTAG, FEATS, HEAD, DEPREL, PHEAD and PDEPREL. Test files contained only the first six, and systems predicted HEAD and DEPREL. Each token has exactly one head, and secondary relations were dropped.
Test sets were targeted at about 5,000 scoring tokens per language, without cutting sentences. Because the sizes are equal, the overall ranking can be computed on the concatenation of all test sets.
McDonald et al. and Nivre et al. had very similar overall scores and both exceeded 89% recall on root tokens. Their root precision differed sharply: 91.07 versus 80.91.
Accuracy varied by part of speech:
- German infinitival markers (PTKZU) were parsed perfectly by every participant.
- Articles typically scored in the nineties.
- Prepositions mostly scored in the fifties or sixties.
- Conjunctions dropped to the forties for Arabic and Dutch.

Interesting ideas

Optional projective layer. Besides the true, possibly non-projective HEAD, the format can carry a guaranteed-projective PHEAD. One file can then serve both parsers that require projective trees and those that do not. This is useful for any treebank meant for mixed toolchains.
Punctuation defined by Unicode category. A token is excluded from scoring when all of its characters have the Unicode Punctuation property. This gives a language-independent rule instead of per-treebank lists, though edge cases such as "_" or "%" also count as punctuation.
Equal-sized test sets. Using a roughly fixed number of scoring tokens per language lets a single pooled score rank systems fairly across languages. It also leaves most of each small treebank available for training.
Error breakdowns beyond LAS. Splitting results by root precision/recall and by part of speech exposes real differences between systems with near-identical LAS. It also shows which constructions, such as prepositions and conjunctions, remain hard. This kind of diagnostic is worth adding to any parser or tagger benchmark.
Report source errors instead of silently fixing them. Conversion errors were not corrected manually. They were reported back to the treebank providers, which keeps the shared data faithful to its source while still improving the original resources over time.

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1596276.1596305

@inproceedings{buchholz2006conll,
  title = {{CoNLL-X Shared Task on Multilingual Dependency Parsing}},
  author = {Buchholz, Sabine and Marsi, Erwin},
  booktitle = {Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)},
  year = {2006},
  doi = {10.3115/1596276.1596305},
  url = {https://doi.org/10.3115/1596276.1596305}
}

References

60 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

A. Arun and F. Keller. 2005. Lexicalization in crosslinguistic probabilistic parsing: The case of French. In Proc. of the 43rd Annual Meeting of the ACL. find
D. Bikel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proc. of the Human Language Technology Conf. (HLT). find
E. Black, S. Abney, D. Flickenger, et al. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California. find
E. Black, F. Jelinek, J. Lafferty, D. Magerman, R. Mercer, and S. Roukos. 1993. Towards history-based grammars: Using richer models for probabilistic parsing. In Proc. of the 31rd Annual Meeting of the ACL. find
S. Buchholz and D. Green. 2006. Quality control of treebanks: documenting, converting, patching. In LREC 2006 workshop on Quality assurance and quality measurement for language and speech resources. 34 For the Swedish Talbanken05 corpus, that work has been done after the shared task (see the treeban… find
A. Chanev, K. Simov, P. Osenova, and S. Marinov. 2006. Dependency conversion and parsing of the BulTreeBank. In Proc. of the LREC-Workshop Merging and Layering Linguistic Information. find
Y. Cheng, M. Asahara, and Y. Matsumoto. 2005. Chinese deterministic dependency analyzer: Examining effects of global features and root node finder. In Proc. of SIGHAN-2005. find
Y.J. Chu and T.H. Liu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396–1400. find
M. Collins, J. Hajic, L. Ramshaw, and C. Tillmann. 1999. A statistical parser for Czech. In Proc. of the 37th Annual Meeting of the ACL. find
M. Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proc. of the 34th Annual Meeting of the ACL. find
M. Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proc. of the 35th Annual Meeting of the ACL. find
M. Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find
B. Cowan and M. Collins. 2005. Morphology and reranking for the statistical parsing of Spanish. In Proc. of the Joint Conf. on Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP). find
D. Csendes, J. Csirik, and T. Gyimóthy. 2004. The Szeged corpus: a POS tagged and syntactically annotated Hungarian natural language corpus. In Proc. of the 5th Intern. Workshop on Linguistically Interpreteted Corpora (LINC). find
W. Daelemans, J. Zavrel, P. Berck, and S. Gillis. 1996. MBT: A memory-based part of speech tagger-generator. In Proc. of the 4th Workshop on Very Large Corpora (VLC). find
A. Dubey and F. Keller. 2003. Probabilistic parsing for German using sister-head dependencies. In Proc. of the 41st Annual Meeting of the ACL. find
J. Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards, 71B:233–240. find
J. Einarsson. 1976. Talbankens skriftspråkskonkordans. find
J. Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In Proc. of the 16th Intern. Conf. on Computational Linguistics (COLING), pages 340– 345. find our card
D. Elworthy. 2000. A finite-state parser with dependency structure output. In Proc. of the 6th Intern. Workshop on Parsing Technologies (IWPT). find
H. Gaifman. 1965. Dependency systems and phrase-structure systems. Information and Control, 8:304–337. find
D. Hays. 1964. Dependency theory: A formalism and some observations. Language, 40:511–525. find
R. Hudson. 1984. Word Grammar. Blackwell. find
T. Kudo and Y. Matsumoto. 2000. Japanese dependency structure analysis based on support vector machines. In Proc. of the Joint Conf. on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC). find
S. Kurohashi and M. Nagao. 1994. KN parser: Japanese dependency/case structure analyzer. In Proceedings of the Workshop on Sharable Natural Language, pages 48–55. find
S. Kurohashi and M. Nagao. 1997. Kyoto University text corpus project. In Proc. of the 5th Conf. on Applied Natural Language Processing (ANLP), pages 115–118. find
Y. Lepage, S. Ando, S. Akamine, and H. Iida. 1998. An annotated corpus in Japanese using Tesnière’s structural syntax. In ACL-COLING Workshop on Processing of Dependency-Based Grammars, pages 109–115. find
D. Lin. 1995. A dependency-based method for evaluating broad-coverage parsers. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI). find
D. Magerman. 1995. Statistical decision-tree models for parsing. In Proc. of the 33rd Annual Meeting of the ACL, pages 276–283. find
M. Marcus, G. Kim, M. Marcinkiewicz, R. Mac-Intyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger. 1994. The Penn treebank: Annotating predicate argument structure. In Proc. of the Workshop on Human Language Technology (HLT). find our card
S. Marinov and J. Nivre. 2005. A data-driven dependency parser for Bulgarian. In Proc. of the 4th Workshop on Treebanks and Linguistic Theories (TLT), pages 89–100. find
R. McDonald and F. Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proc. of the 11th Conf. of the European Chapter of the ACL (EACL). find our card
R. McDonald, K. Crammer, and F. Pereira. 2005a. Online large-margin training of dependency parsers. In Proc. of the 43rd Annual Meeting of the ACL. find our card
R. McDonald, F. Pereira, K. Ribarov, and J. Hajic. 2005b. Non-projective dependency parsing using spanning tree algorithms. In Proc. of the Joint Conf. on Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP). find our card
I. Mel’čuk. 1988. Dependency Syntax: Theory and Practice. The SUNY Press, Albany, N.Y. find
J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. of the 43rd Annual Meeting of the ACL, pages 99–106. find
J. Nivre and M. Scholz. 2004. Deterministic dependency parsing of English text. In Proc. of the 20th Intern. Conf. on Computational Linguistics (COLING), pages 64–70. find our card
J. Nivre, J. Hall, and J. Nilsson. 2004. Memory-based dependency parsing. In Proc. of the 8th Conf. on Computational Natural Language Learning (CoNLL), pages 49–56. find
K. Oflazer. 1999. Dependency parsing with an extended finite state approach. In Proc. of the 37th Annual Meeting of the ACL, pages 254–260. find
G. Sampson. 1995. English for the Computer: The SUSANNE Corpus and analytic scheme. Clarendon Press. find
K. Sima’an, A. Itai, Y. Winter, A. Altman, and N. Nativ. 2001. Building a tree-bank of modern Hebrew text. In Journal Traitement Automatique des Langues (t.a.l.) — Special Issue on Natural Language Processing and Corpus Linguistics. find
D. Sleator and D. Temperley. 1993. Parsing English with a link grammar. In Proc. of the 3rd Intern. Workshop on Parsing Technologies (IWPT). find our card
P. Tapanainen and T. Järvinen. 1997. A non-projective dependency parser. In Proc. of the 5th Conf. on Applied Natural Language Processing (ANLP). find
U. Teleman, 1974. Manual för grammatisk beskrivning av talad och skriven svenska (MAMBA). find
L. Tesnière. 1959. Elément de syntaxe structurale. Klincksieck, Paris. find
H. Yamada and Y. Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proc. of the 8th Intern. Workshop on Parsing Technologies (IWPT), pages 195– 206. find our card
Papers by participants of CoNLL-X (this volume) find
G. Attardi. 2006. Experiments with a multilanguage non-projective dependency parser. find
E. Bick. 2006. LingPars, a linguistically inspired, language-independent machine learner for dependency treebanks. find
S. Canisius, T. Bogers, A. van den Bosch, J. Geertzen, and E. Tjong Kim Sang. 2006. Dependency parsing by inference over high-recall dependency predictions. find
M. Chang, Q. Do, and D. Roth. 2006. A pipeline model for bottom-up dependency parsing. find
S. Corston-Oliver and A. Aue. 2006. Dependency parsing with reference to Slovene, Spanish and Swedish. find
M. Dreyer, D. Smith, and N. Smith. 2006. Vine parsing and minimum risk reranking for speed and precision. find
R. Johansson and P. Nugues. 2006. Investigating multilingual dependency parsing. find
R. McDonald, K. Lerman, and F. Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. find
J. Nivre, J. Hall, J. Nilsson, G. Eryiǧit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector machines. find
S. Riedel, R. Çakıcı, and I. Meza-Ruiz. 2006. Multi-lingual dependency parsing with incremental integer linear programming. find
M. Schiehlen and K. Spranger. 2006. Language independent probabilistic context-free parsing bolstered by machine learning. find
N. Shimizu. 2006. Maximum spanning tree algorithm for non-projective labeled dependency parsing. find
D. Yuret. 2006. Dependency parsing as a classication problem. find