Elamur Research⌕
☰
home / papers / 2021 / A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021

A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021

Summary

The authors compare modern dependency parsers on two criteria at once: parsing quality and resource cost. They take one representative from each of three approaches: a transition-based parser built on a pointer network (L2R), the graph-based Biaffine, and a parser that reduces parsing to sequence labeling (SeqLab). All three systems are built on BiLSTMs, use the same software and hardware, and run without large pretrained language models. The authors vary the number of BiLSTM layers and units, train many configurations on four typologically diverse treebanks, and build Pareto fronts. Some plots set LAS against parsing speed on GPU and CPU, and others set it against the energy consumed in training. On speed, only Biaffine and SeqLab make up the front: the former is more accurate, while the latter is noticeably faster. L2R matches Biaffine in accuracy but is considerably slower. SeqLab is the least energy-efficient to train. The overall conclusion is that Biaffine is a balanced default choice, while SeqLab is worth choosing when parsing speed matters most. For practitioners choosing a parser on a limited budget, this is a useful and honest snapshot of the state of the field.

Key points

  • Data: UD-Hindi-HDTB, UD-Polish-PDB, UD-Korean-Kaist and the Chinese Penn Treebank. Together they cover 3 language families and 4 subfamilies, with manual or manually corrected annotations.
  • Model inputs: FastText embeddings reduced to 100 dimensions with PCA (different embeddings were used for Chinese), plus character embeddings. Only the BiLSTM size was varied. Hardware: Intel Core i7-7700 and GeForce GTX 1080.
  • Training energy was measured on GPU and CPU with the energyusage package and reported in joules. It depends almost linearly on the time to convergence. The authors found no direct relationship between the energy consumed and LAS. L2R is especially sensitive to the particular treebank.
  • The authors also ran the parsers locally on the English PTB. Biaffine parses about 1000 sentences/s on GPU (LAS 94.07), Pointer-LR about 95 sentences/s (LAS 94.47), and SeqLab about 1064 sentences/s (LAS 91.49).
  • The code, with minor modifications, is published on the LyS group website (grupolys.org). The limitations the authors name include the small sample of languages, no analysis of features (in particular POS tags), and only one model per configuration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.iwpt-1.12 · arXiv 2106.04216

@inproceedings{anderson2021modest,
  title = {{A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021}},
  author = {Anderson, Mark and Gómez-Rodríguez, Carlos},
  booktitle = {Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021)},
  year = {2021},
  doi = {10.18653/v1/2021.iwpt-1.12},
  eprint = {2106.04216},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.iwpt-1.12}
}

References

59 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Anderson and Carlos Gómez-Rodrı́guez. 2020a. Distilling neural networks for greener and faster dependency parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 2–13, Online. Asso… find our card
  2. Mark Anderson and Carlos Gómez-Rodrı́guez. 2020b. On the frailty of universal POS tags for neural UD parsers. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 69–96, Online. Association for Computational Linguistics. find our card
  3. Riyaz Ahmad Bhat, Rajesh Bhatt, Annahita Farudi, Prescott Klassen, Bhuvana Narasimhan, Martha Palmer, Owen Rambow, Dipti Misra Sharma, Ashwini Vaidya, Sri Ramagurumurthy Vishnu, et al. 2017. The hindi/urdu treebank project. In Handbook of Linguistic Annotation, pages 659–697. Springer. find
  4. Federica Cognola and Jan Casalicchio. 2018. On the null-subject phenomenon. Null subjects in Generative Grammar. A Synchronic and Diachronic Perspective, pages 1–28. find
  5. Bernard Comrie. 1978. Ergativity. In Winfred P. Lehmann, editor, Syntactic Typology: Studies in the Phenomenology of Language, pages 329–394. University of Texas Press, Austin. find
  6. Mathieu Dehouck, Mark Anderson, and Carlos Gómez-Rodrı́guez. 2020. Efficient EUD parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 192–205, Online. Association for Computational L… find
  7. Dana Bielec. 1998. Polish: An Essential Grammar. Routledge, London. find
  8. Timothy Dozat and Christopher D Manning. 2017. Deep biaffine attention for neural dependency parsing. Proceedings of the 5th International Conference on Learning Representations. find our card
  9. BIPM. 2019. Le Système international d’unités / The International System of Units (‘The SI Brochure’), ninth edition. Bureau international des poids et mesures. find
  10. Jack Edmonds. 1967. Optimum branchings. Journal of Research of the national Bureau of Standards B, 71(4):233–240. find
  11. Ronald F Feldstein. 2001. A concise Polish grammar. Citeseer. find
  12. Lucien Brown. 2015. Honorifics and politeness. The handbook of Korean linguistics, pages 303–319. find
  13. Suk-Jin Chang. 1996. Philadelphia. find
  14. Korean. John Benjamins, Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar. Association for Computational Linguistics. find our card
  15. Key-Sun Choi, Young S Han, Young G Han, and Oh W Kwon. 1994. Kaist tree bank project for korean: Present and future development. In Proceedings of the International Workshop on Sharable Natural Language Resources, pages 7–14. Citeseer. find
  16. Yoeng-Jin Chu and Tseng-Hong Liu. 1965. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396–1400. find
  17. Jayeol Chun, Na-Rae Han, Jena D Hwang, and Jinho D Choi. 2018. Building universal dependency treebanks in korean. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  18. Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914– 1925, Brussels, Belgium. Association for Computational Linguist… find our card
  19. Daniel Fernández-González and Carlos Gómez-Rodrı́guez. 2019. Left-to-right dependency parsing with pointer networks. In Proceedings of NAACL-HLT, pages 710–716. find
  20. Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2018. SUD or surface-syntactic Universal Dependencies: An annotation scheme near-isomorphic to UD. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 66–74, Brussels, Belgium. Association for Computational L… find
  21. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find our card
  22. One-Soon Her and Chen-Tien Hsieh. 2010. On the semantic distinction between classifiers and measure words in chinese. Language and linguistics, 11(3):527–551. find
  23. Magnus Jacobsen, Mikkel H. Sørensen, and Leon Derczynski. 2021. Optimal size-performance tradeoffs: Weighing pos tagger models. find
  24. Tao Ji, Yuanbin Wu, and Man Lan. 2019. Graph-based dependency parsing with graph neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2475–2485, Florence, Italy. Association for Computational Linguistics. find
  25. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. find our card
  26. HE Klockmann. 2012. Polish numerals and quantifiers: A syntactic analysis of subject-verb agreement mismatches. Master’s thesis, Utrecht University. find
  27. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one MST parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1744–1753, Austin, Texas. Association fo… find
  28. Iksop Lee and Robert Ramsey. 2000. The Korean Language. State University of New York Press, New York. find
  29. Charles N. Li and Sandra A. Thompson. 1981. Mandarin Chinese: a Functional Reference Grammar. University of California Press, Berkeley. find
  30. Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018a. Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 138–143. Association for Computati… find
  31. Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao. 2018b. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3203–3214, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  32. Meichun Liu. 2015. Tense and aspect in mandarin chinese. The Oxford Handbook of Chinese Linguistics, pages 274–289. find
  33. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403–1414. find
  34. Mitchell P Marcus and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2). find our card
  35. R. S. McGregor. 1977. Outline of Hindi Grammar. Oxford University Press, Delhi. 2nd edition. find
  36. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülşen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135. find
  37. Martha Palmer, Rajesh Bhatt, Bhuvana Narasimhan, Owen Rambow, Dipti Misra Sharma, and Fei Xia. 2009. Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure. In The 7th International Conference on Natural Language Processing, pages 14–17. find
  38. Waltraud Paul. 2008. The serial verb construction in chinese: A tenacious myth and a gordian knot. Linguistic Review - LINGUIST REV, 25:367–411. find
  39. G. J. Ramstedt. 1968. A Korean Grammar, volume 82 of Memoirs of the Finno-Ugric Society. Anthropological Publications, Oosterhout, The Netherlands. find
  40. James Robertson. 2006. The Testament of Gideon Mack. Penguin Books Ltd. find
  41. Anna Siewierska. 1993. Syntactic weight vs. information structure and word order variation in polish. Journal of Linguistics, 29:233–265. find
  42. Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82 treebanks, 34 models: Universal Dependency parsing with multi-treebank models. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 113–123, Br… find
  43. Aaron Smith, Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2018. An investigation of the interactions between pre-trained word embeddings, character models and POS tags in dependency parsing. In EMNLP 2018: 2018 Conference on Empirical Methods in Natural Language Processing, pages 2711– 2720. find
  44. R. Snell and S. Weightman. 1989. Teach yourself Hindi. Hodder and Stoughton, London. find
  45. Ho-Min Sohn. 1999. The Korean Language. Cambridge University Press, Cambridge. find
  46. Seok Choong Song. 1988. Explorations in Korean Syntax and Semantics. Institute of East Asian Studies, University of California Berkeley, Berkeley. find
  47. Drahomı́ra Johanka Spoustová and Miroslav Spousta. 2010. Dependency parsing as a sequence labeling task. The Prague Bulletin of Mathematical Linguistics, 94(2010):7–14. find
  48. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find our card
  49. Michalina Strzyz, David Vilares, and Carlos Gómez-Rodrı́guez. 2019. Viable dependency parsing as sequence labeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pa… find
  50. Michalina Strzyz, David Vilares, and Carlos Gómez-Rodrı́guez. 2020. Bracketing encodings for 2-planar dependency parsing. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2472–2484. find
  51. Bernd Wiese. 2011. Optimal specifications: On case marking in polish. Syntax and morphology multidimensional, 24:101. find
  52. Alina Wróblewska. 2018. Extended and enhanced polish dependency bank in universal dependencies format. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 173–182. find
  53. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207. find
  54. Nianwen Xue, Fu-Dong Chiou, and Martha Palmer. 2002. Building a large-scale annotated chinese corpus. In COLING 2002: The 19th International Conference on Computational Linguistics. find
  55. Xuan Zhang and Kevin Duh. 2020. Reproducible and efficient benchmarks for hyperparameter optimization of neural machine translation systems. Transactions of the Association for Computational Linguistics, 8:393–408. find
  56. Yu Zhang, Zhenghua Li, and Min Zhang. 2020a. Efficient second-order TreeCRF for neural dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3295–3305, Online. Association for Computational Linguistics. find our card
  57. Yu Zhang, Zhenghua Li, Houquan Zhou, and Min Zhang. 2020b. Is POS tagging necessary or even helpful for neural dependency parsing? arXiv preprint arXiv:2003.03204. — [arXiv:2003.03204](https://arxiv.org/abs/2003.03204) find
  58. Junru Zhou and Hai Zhao. 2019. Head-driven phrase structure grammar parsing on Penn Treebank. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2396– 2408. find
  59. Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. 2021. HULK: An energy efficiency benchmark platform for responsible natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 3… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anderson-2021-modest-pareto-optimisation-analysis-dependency/