📄 LIMSI@CoNLL'17: UD Shared Task

Summary

The paper describes the LIMSI team's system for the CoNLL 2017 UD Shared Task. This is a multilingual dependency parsing competition that goes from raw text to Universal Dependencies annotation. The authors focused on small treebanks with fewer than 1,000 training sentences. For these, they proposed a cascaded combination of several base parsers: monolingual, delexicalized cross-lingual, and trained on trees transferred through parallel corpora. Each parser is assigned a "competence zone" in advance, i.e., the subset of tokens it annotates. The parser takes into account the partial tree produced by the preceding stages, so each token is handled by exactly one parser. The best strategy for each language was selected on the dev data. On several small treebanks this yielded a noticeable gain over UDPipe. However, the overall result fell below the baseline because of sharp failures on certain languages. Analysis of these failures revealed substantial annotation discrepancies between treebanks of the same language. This is an important caveat for anyone combining several UD corpora of the same language.

Key points

• The system's official result is 67.72 LAS, ranking 17th out of 33 participants. The UDPipe 1.1 baseline obtained 68.35 LAS and ranked 13th. Had the same models as the baseline been submitted for the problematic treebanks, the system would have scored 68.90 LAS and ranked 9th. In the unofficial post-evaluation ranking it is 12th.
• The core of the system is the authors' own parser, PanParser. It is transition-based: it uses ArcEager, an averaged perceptron, dynamic oracles, and a beam of size 8 or greedy search. PanParser can learn from partial trees and build parses subject to constraints, and this is what the cascade relies on. The UDPipe models in the cascade are not retrained; their output is only filtered.
• The source language for cross-lingual transfer is chosen among treebanks with more than 2,000 sentences. They are first transformed with rules based on WALS, and then the one with the lowest KL cpos3 divergence is selected. Parallel data are taken from OPUS, aligned with fast_align, and partial projection following Lacroix et al. (2016) is applied.
• The authors attribute the failures on the PUD treebanks to the fact that different corpora of the same language apply the annotation scheme differently. Moreover, in some corpora the "raw" text has already been partially preprocessed. In Russian-SynTagRus, Finnish-FTB and Greek, multiword tokens are already marked. Danish and Finnish-FTB are provided already tokenized.
• The authors acknowledge the method's limitations: there are no confidence estimation mechanisms, delexicalized models are unreliable, and there are too few dev data to determine the competence zones accurately.