Elamur Research⌕
☰
home / papers / 2007 / The CoNLL 2007 Shared Task on Dependency Parsing

The CoNLL 2007 Shared Task on Dependency Parsing

Summary

This paper reports on the CoNLL 2007 shared task, which continued the 2006 work on data-driven dependency parsing and added a second track. In the multilingual track, one parsing system had to learn from treebanks covering ten typologically varied languages, namely Chinese, Arabic, Turkish, Hungarian, Basque, Czech, Greek, Catalan, Italian and English. The domain adaptation track asked participants to train on Wall Street Journal data and parse English text from other domains without any labeled target-domain data. The organizers converted existing treebanks into a shared ten-column tab-separated format, with as little conversion as possible. They scored systems mainly by LAS, counting punctuation, and also described the participants' approaches and analyzed the results. Twenty-three systems entered the multilingual track and ten entered the adaptation track. In adaptation, systems limited to the supplied resources (closed class) beat systems allowed extra resources (open class) on chemical abstracts. The task matters because it created a large, reproducible benchmark and released all system outputs for comparative error analysis.

Key points

  • Data format: The task used the same ten-field column format as 2006 (ID, FORM, LEMMA, CPOSTAG, POSTAG, FEATS, HEAD, DEPREL, PHEAD, PDEPREL), encoded in UTF-8. The first six fields are parser input, and HEAD/DEPREL are the predicted output.
  • Evaluation: LAS was the official metric, with UAS and label accuracy also reported. Unlike 2006, every token counted, including punctuation. The scoring script is eval07.pl.
  • Data size: Training sets were capped at 50,000–500,000 tokens so that all teams could train on all the data. For example, PADT (Arabic) grew from about 54,000 to about 112,000 tokens.
  • Domain adaptation setup: The source was WSJ, with biomedical abstracts as development data and chemical abstracts and CHILDES parent-child dialogues as test sets. Large unlabeled corpora were provided for each, and systems competed in closed and open classes.
  • Adaptation results:
    • The best chemical-domain LAS was 81.06 (Sagae and Tsujii), compared with 89.01 for the same team on English in the multilingual track. Roughly a third of the chemical test words were unseen.
    • On CHILDES, the top UAS was only 62.49, mostly because auxiliary and main verbs were annotated differently than in the WSJ data. Submitting CHILDES results was therefore made optional.

Interesting ideas

  • I1. Cap training set size. Limiting how much training data each language gets lets every participant use all of it within the time available, so comparisons stay fair. This is a useful rule for any benchmark run under a deadline.
  • I2. Separate closed and open conditions. Splitting results by allowed resources shows whether gains come from the method or from extra data. Open-class resources were also barred from the test domains to avoid leakage.
  • I3. Check annotation conventions across domains. The CHILDES result shows that scores in a "new domain" can be dominated by differences in annotation guidelines rather than in the language itself. Datasets should be checked for convention mismatches before drawing conclusions about domain shift.
  • I4. Publish every system's output. Releasing all outputs for all datasets turns a one-off competition into a lasting resource for error analysis across languages and systems.
  • I5. Treat output-format mapping as its own problem. Existing parsers did poorly in the open class mainly because their native output had to be converted to the task's annotation scheme. Learning such mappings could let teams reuse existing tools without losing much accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dependency parsing · Treebanks

Cited by

353 papers of our library cite it (of the 140884 whose bibliographies we read); 45 of them have a page here, 308 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nivre2007conll,
  title = {{The CoNLL 2007 Shared Task on Dependency Parsing}},
  author = {Nivre, Joakim and Hall, Johan and Kübler, Sandra and McDonald, Ryan and Nilsson, Jens and Riedel, Sebastian and Yuret, Deniz},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1096/}
}

References

57 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Abeillé, editor. 2003. Treebanks: Building and Using Parsed Corpora. Kluwer. find
  2. I. Aduriz, M. J. Aranzabe, J. M. Arriola, A. Atutxa, A. Diaz de Ilarraza, A. Garmendia, and M. Oronoz. 2003. Construction of a Basque dependency treebank. In Proc. of the 2nd Workshop on Treebanks and Linguistic Theories (TLT). find
  3. G. Attardi, F. Dell’Orletta, M. Simi, A. Chanev, and M. Ciaramita. 2007. Multilingual dependency parsing and domain adaptation using desr. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  4. E. Bick. 2007. Hybrid ways to improve domain independence in an ML dependency parser. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  5. J. Blitzer, R. McDonald, and F. Pereira. 2006. Domain adaptation with structural correspondence learning. In Proc. of the Conf. on Empirical Methods in Natural Language Processing (EMNLP). find our card
  6. A. Böhmová, J. Hajič, E. Hajičová, and B. Hladká. 2003. The PDT: a 3-level annotation scenario. In Abeillé (2003), chapter 7, pages 103–127. find
  7. R. Brown. 1973. A First Language: The Early Stages. Harvard University Press. find
  8. S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proc. of the Tenth Conf. on Computational Natural Language Learning (CoNLL). find our card
  9. S. Canisius and E. Tjong Kim Sang. 2007. A constraint satisfaction approach to dependency parsing. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  10. X. Carreras. 2007. Experiments with a high-order projective dependency parser. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  11. E. Charniak. 2000. A maximum-entropy-inspired parser. In Proc. of the First Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  12. C. Chelba and A. Acero. 2004. Adaptation of maximum entropy capitalizer: Little data can help a lot. In Proc. of the Conf. on Empirical Methods in Natural Language Processing (EMNLP). find
  13. K. Chen, C. Luo, M. Chang, F. Chen, C. Chen, C. Huang, and Z. Gao. 2003. Sinica treebank: Design criteria, representational issues and implementation. In Abeillé (2003), chapter 13, pages 231–248. find
  14. W. Chen, Y. Zhang, and H. Isahara. 2007. A two-stage parser for multilingual dependency parsing. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  15. M. Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proc. of the 35th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  16. M. A. Covington. 2001. A fundamental algorithm for dependency parsing. In Proc. of the 39th Annual ACM Southeast Conf. find
  17. D. Csendes, J. Csirik, T. Gyimóthy, and A. Kocsor. 2005. The Szeged Treebank. Springer. find
  18. H. Daumé and D. Marcu. 2006. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101–126. find
  19. M. Dredze, J. Blitzer, P. P. Talukdar, K. Ganchev, J. Graca, and F. Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find our card
  20. X. Duan, J. Zhao, and B. Xu. 2007. Probabilistic parsing action models for multi-lingual dependency parsing. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  21. G. Eryiğit. 2007. ITU validation set for Metu-Sabancı Turkish Treebank. URL: http://www3.itu.edu.tr/∼gulsenc/papers/validationset.pdf. ↗
  22. R. Florian, H. Hassan, A. Ittycheriah, H. Jing, N. Kambhatla, A. Luo, N. Nicolov, and S. Roukos. 2004. A statisical model for multilingual entity detection and tracking. In Proc. of the Human Language Technology Conf. and the Annual Meeting of the North American Chapter of the Association for Comput… find
  23. D. Gildea. 2001. Corpus variation and parser performance. In Proc. of the Conf. on Empirical Methods in Natural Language Processing (EMNLP). find our card
  24. J. Hajič, O. Smrž, P. Zemánek, J. Šnaidauf, and E. Beška. 2004. Prague Arabic dependency treebank: Development in data and tools. In Proc. of the NEMLAR Intern. Conf. on Arabic Language Resources and Tools. find
  25. J. Hall, J. Nilsson, J. Nivre, G. Eryiğit, B. Megyesi, M. Nilsson, and M. Saers. 2007a. Single malt or blended? A study in multilingual parser optimization. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  26. K. Hall, J. Havelka, and D. Smith. 2007b. Log-linear models of non-projective trees, k-best MST parsing and tree-ranking. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  27. R. Johansson and P. Nugues. 2007a. Extended constituent-to-dependency conversion for English. In Proc. of the 16th Nordic Conf. on Computational Linguistics (NODALIDA). find our card
  28. R. Johansson and P. Nugues. 2007b. Incremental dependency parsing using online learning. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  29. T. Kudo and Y. Matsumoto. 2002. Japanese dependency analysis using cascaded chunking. In Proc. of the Sixth Conf. on Computational Language Learning (CoNLL). find
  30. S. Kulick, A. Bies, M. Liberman, M. Mandel, R. Mc-Donald, M. Palmer, A. Schein, and L. Ungar. 2004. Integrated annotation for biomedical information extraction. In Proc. of the Human Language Technology Conf. and the Annual Meeting of the North American Chapter of the Association for Computational L… find
  31. B. MacWhinney. 2000. The CHILDES Project: Tools for Analyzing Talk. Lawrence Erlbaum. find
  32. P. R. Mannem. 2007. Online learning for deterministic dependency parsing. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  33. M. Marcus, B. Santorini, and M. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  34. S. Marinov. 2007. Covington variations. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  35. M. A. Martı́, M. Taulé, L. Màrquez, and M. Bertran. 2007. CESS-ECE: A multilingual and multilevel annotated corpus. Available for download from: http://www.lsi.upc.edu/∼mbertran/cess-ece/. ↗
  36. D. McClosky, E. Charniak, and M. Johnson. 2006. Reranking and self-training for parser adaptation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics. find our card
  37. R. McDonald and J. Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proc. of the Joint Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find our card
  38. R. McDonald and F. Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proc. of the 11th Conf. of the European Chapter of the Association for Computational Linguistics (EACL). find our card
  39. R. McDonald, F. Pereira, K. Ribarov, and J. Hajič. 2005. Non-projective dependency parsing using spanning tree algorithms. In Proc. of the Human Language Technology Conf. and the Conf. on Empirical Methods in Natural Language Processing (HLT/EMNLP). find our card
  40. S. Montemagni, F. Barsotti, M. Battista, N. Calzolari, O. Corazzari, A. Lenci, A. Zampolli, F. Fanciulli, M. Massetani, R. Raffaelli, R. Basili, M. T. Pazienza, D. Saracino, F. Zanzotto, N. Nana, F. Pianesi, and R. Delmonte. 2003. Building the Italian Syntactic-Semantic Treebank. In Abeillé (2003),… find
  41. T. Nakagawa. 2007. Multilingual dependency parsing using Gibbs sampling. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  42. L.-M. Nguyen, T.-P. Nguyen, and A. Shimazu. 2007. A multilingual dependency analysis system using online passive-aggressive learning. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  43. J. Nivre, J. Hall, J. Nilsson, A. Chanev, G. Eryiğit, S. Kübler, S. Marinov, and E. Marsi. 2007. Malt-Parser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13:95–135. find
  44. K. Oflazer, B. Say, D. Zeynep Hakkani-Tür, and G. Tür. 2003. Building a Turkish treebank. In Abeillé (2003), chapter 15, pages 261–277. find
  45. P. Prokopidis, E. Desypri, M. Koutsombogera, H. Papageorgiou, and S. Piperidis. 2005. Theoretical and practical issues in the construction of a Greek dependency treebank. In Proc. of the 4th Workshop on Treebanks and Linguistic Theories (TLT). find
  46. S. Riedel, R. Çakıcı, and I. Meza-Ruiz. 2006. Multilingual dependency parsing with incremental integer linear programming. In Proc. of the Tenth Conf. on Computational Natural Language Learning (CoNLL). find
  47. B. Roark and M. Bacchiani. 2003. Supervised and unsupervised PCFG adaptation to novel domains. In Proc. of the Human Language Technology Conf. and the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL). find
  48. K. Sagae and A. Lavie. 2006. Parser combination by reparsing. In Proc. of the Human Language Technology Conf. of the North American Chapter of the Association of Computational Linguistics (HLT/NAACL). find our card
  49. K. Sagae and J. Tsujii. 2007. Dependency parsing and domain adaptation with LR models and parser ensembles. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find our card
  50. M. Schiehlen and Kristina Spranger. 2007. Global learning of labelled dependency trees. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  51. G. Schneider, K. Kaljurand, F. Rinaldi, and T. Kuhn. 2007. Pro3Gres parser in the CoNLL domain adaptation shared task. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  52. N. Shimizu and H. Nakagawa. 2007. Structural correspondence learning for dependency parsing. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  53. I. Titov and J. Henderson. 2006. Porting statistical parsers with data-defined kernels. In Proc. of the Tenth Conf. on Computational Natural Language Learning (CoNLL). find
  54. I. Titov and J. Henderson. 2007. Fast and robust multilingual dependency parsing with a generative latent variable model. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  55. R. Watson and T. Briscoe. 2007. Adapting the RASP system for the CoNLL07 domain-adaptation task. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  56. Y.-C. Wu, J.-C. Yang, and Y.-S. Lee. 2007. Multilingual deterministic dependency parsing framework using modified finite Newton method support vector machines. In Proc. of the CoNLL 2007 Shared Task. EMNLP-CoNLL. find
  57. H. Yamada and Y. Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proc. 8th International Workshop on Parsing Technologies (IWPT). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nivre-2007-conll-2007-shared-task-dependency/