Elamur Research⌕
☰
home / papers / 2016 / Multi-lingual Dependency Parsing Evaluation: a Large-scale Analysis of Word Order Properties using Artificial Data

Multi-lingual Dependency Parsing Evaluation: a Large-scale Analysis of Word Order Properties using Artificial Data

Paper · 2016 · Kristina Gulordava and Paola Merlo

Summary

Comparing dependency parsers across languages is hard because accuracy differences may come from treebank size, sentence length, annotation choices, or the languages' own grammar, and these factors are tangled together. Gulordava and Merlo isolate the effect of word order with an interventionist design. They build artificial treebanks by reordering the words of each sentence while leaving the unordered dependency tree, the vocabulary and the train/test split unchanged. Each original and permuted treebank then forms a minimal pair that differs only in linear order. Two kinds of reordering are applied: one shortens dependencies, and the other reduces how variable head–child ordering is. The authors measure these properties with the DLM ratio and arc direction entropy, then compare UAS and LAS on original and permuted data for fourteen treebanks covering twelve languages. Longer dependencies and freer word order both lower parsing accuracy, and word order variability has the larger effect. The same pairing also allows sentence-by-sentence error analysis, giving a controlled way to separate linguistic from extra-linguistic causes of parsing difficulty.

Key points

  • Data: 14 treebanks spanning 12 languages: English, French, Spanish, Italian, Polish, Bulgarian, Finnish, Dutch, German, Persian, Latin (two treebanks, Vulgate and Cicero) and Ancient Greek (two treebanks, New Testament and Herodotus). The Latin and Ancient Greek texts come from the PROIEL treebanks.
  • Word order measures:
    • Arc direction entropy is the left/right head–child order entropy, conditioned on the dependency label and the PoS tags of head and child, and weighted by tuple frequency.
    • The DLM ratio compares actual dependency length with the minimal projective length, normalised by squared sentence length.
  • Evaluation: parsers are trained and tested on original vs. permuted versions with identical splits. Accuracy gaps (UAS/LAS) can therefore be attributed to word order alone.
  • Main finding: higher word order entropy hurts parsing more strongly than longer dependencies. This confirms earlier small-scale observations (e.g., McDonald and Nivre 2011) at a larger scale.
  • Side result: across treebanks, the DLM ratio correlates with the share of non-projective arcs (Pearson 0.66, p < 0.02). Non-projectivity is named as the next property to study.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GenevaGeneva · Switzerland29 works · 3,158 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksDependency parsing · 131 works

Cite

Cite as Kristina Gulordava and Paola Merlo. 2016. Multi-lingual Dependency Parsing Evaluation: a Large-scale Analysis of Word Order Properties using Artificial Data. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00103

@article{gulordava2016multi,
  title = {{Multi-lingual Dependency Parsing Evaluation: a Large-scale Analysis of Word Order Properties using Artificial Data}},
  author = {Kristina Gulordava and Paola Merlo},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00103},
  url = {https://doi.org/10.1162/tacl_a_00103}
}

References

38 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić, Maria Jesus Aranzabe, Aitziber Atutxa, Cristina Bosco, Jinho Choi, Marie-Catherine de Marneffe, Timothy Dozat, Richárd Farkas, Jennifer Foster, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Yoav Goldberg, Jan Hajič, Anders Trærup Johannsen, Jenna Kanerva, Juha Kuokkala, Veronika Laip… find
  2. Miguel Ballesteros and Joakim Nivre. 2012. MaltOptimizer: An optimization tool for MaltParser. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL ’12), pages 58–62, Avignon, France, April. find
  3. Emily M. Bender, Dan Flickinger, Stephan Oepen, and Yi Zhang. 2011. Parser evaluation over local and non-local deep dependencies in a large corpus. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’11), pages 397– 408, Edinburgh, United Kingdom, July.
  4. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X ’06), pages 149–164, New York City, NY, USA, June.
  5. F. R. K. Chung. 1984. On optimal linear arrangements of trees. Computers & Mathematics with Applications, 10(1):43–60. find
  6. Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D Manning. 2014. Universal Stanford dependencies: A cross-linguistic typology. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC ’1…
  7. Vera Demberg and Frank Keller. 2008. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193–210. find
  8. Ramon Ferrer-i-Cancho and Haitao Liu. 2014. The risks of mixing dependency lengths from sequences of different length. Glottotheory, 5(2):143–155. find
  9. Ramon Ferrer-i-Cancho. 2006. Why do syntactic links not cross? EPL (Europhysics Letters), 76(6):12–28. find
  10. Sandiway Fong and Robert Berwick. 2008. Treebank parsing and knowledge of language: A cognitive perspective. In Proceedings of the 30th Annual Conference of the Cognitive Science Society, pages 539–544, Washington, DC, USA, July. find
  11. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015a. Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33):10336–10341. find
  12. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015b. Quantifying word order freedom in dependency corpora. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 91–100, Uppsala, Sweden, August. find
  13. Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1–76. find
  14. Daniel Gildea and David Temperley. 2010. Do grammars minimize dependency length? Cognitive Science, 34(2):286–310. find
  15. Kristina Gulordava and Paola Merlo. 2015. Diachronic trends in word order freedom and dependency length in dependency-annotated corpora of Latin and Ancient Greek. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 121–130, Uppsala, Sweden, August. K… find
  16. Dag T. T. Haug and Marius L. Jøhndal. 2008. Creating a parallel treebank of the Old Indo-European Bible translations. In Proceedings of the 2nd Workshop on Language Technology for Cultural Heritage Data, pages 27–34, Marrakech, Morocco, June. find
  17. John A. Hawkins. 1994. A performance theory of order and constituency. find
  18. Cambridge University Press, Cambridge, UK. James Henderson, Paola Merlo, Ivan Titov, and Gabriele Musillo. 2013. Multilingual joint parsing of syntactic and semantic dependencies with a latent variable model. Computational Linguistics, 39(4):949–998.
  19. Haitao Liu. 2010. Dependency direction as a means of word-order typology: A method based on dependency treebanks. Lingua, 120(6):1567–1578, June. find
  20. Ryan McDonald and Joakim Nivre. 2011. Analyzing and integrating dependency parsers. Computational Linguistics, 37(1):197–230.
  21. Ryan McDonald, Kevin Lerman, and Fernando Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL’06), pages 216–220, New York, NY, USA, June.
  22. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 62–72, Edinburgh, Scotland, UK, July.
  23. Paola Merlo. 2015. Evaluation of two-level dependency representations of argument structure in long-distance dependencies. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 221–230, Uppsala, Sweden, August. find
  24. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 629–637, Jeju Island, Korea, July.
  25. Joakim Nivre, Johan Hall, and Jens Nilsson. 2006. MaltParser: A data-driven parser-generator for dependency parsing. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC’06), pages 2216–2219, Genova, Italy, May. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan…
  26. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez Rodrı́guez. 2010. Evaluation of dependency parsers on unbounded dependencie…
  27. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the 10th Int…
  28. Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Long papers, pages 351–359, Suntec, Singapore…
  29. Mohammad Sadegh Rasooli, Manouchehr Kouhestani, and Amirsaeid Moloodi. 2013. Development of a Persian syntactic dependency treebank. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 306–314, Atlan… find
  30. Sujith Ravi, Kevin Knight, and Radu Soricut. 2008. Automatic prediction of parser accuracy. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 887–896, Honolulu, Hawaii, October. find
  31. Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP ’09), pages 813–821, Suntec, Singapore, August.
  32. Rudolf Rosa and Zdenek Zabokrtsky. 2015. Klcpos3 -a language similarity measure for delexicalized parser transfer. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Pap… find
  33. Mihai Surdeanu and Christopher D. Manning. 2010. Ensemble models for dependency parsing: Cheap and good? In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 649–652, Los Angeles, California, June. find
  34. David Temperley. 2007. Minimization of dependency length in written English. Cognition, 105(2):300–333. find
  35. Harry Joel Tily. 2010. The Role of Processing Complexity in Word Order Variation and Change. find
  36. Ph.D. Thesis, Stanford University. Ivan Titov and James Henderson. 2007. A latent variable model for generative dependency parsing. In Proceedings of the 10th International Conference on Parsing Technologies, IWPT ’07, pages 144–155, Prague, Czech Republic, June. find
  37. Ivan Titov, James Henderson, Paola Merlo, and Gabriele Musillo. 2009. Online graph planarisation for synchronous parsing of semantic and syntactic dependencies. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI ’09), pages 1562–1567, Pasadena, California, US…
  38. Reut Tsarfaty, Joakim Nivre, and Evelina Andersson. 2011. Evaluating dependency parsing: Robust and heuristics-free cross-annotation evaluation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP ’11), pages 385–396, Edinburgh, Scotland, UK, July. Marcin…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gulordava-2016-multi-lingual-dependency-parsing-evaluation/