Elamur Research⌕
☰
home / papers / 2020 / On the Frailty of Universal POS Tags for Neural UD Parsers

On the Frailty of Universal POS Tags for Neural UD Parsers

Summary

The paper asks whether universal POS (UPOS) tags still help modern neural dependency parsers that already use character-level representations. The authors trained NCRF++ taggers on nine UD v2.4 treebanks and saved checkpoints at controlled accuracy levels, from about 60% up to each tagger's maximum. They fed these predicted tags, gold tags or no tags into two parsers: the graph-based Biaffine and the transition-based UUParser. Further experiments varied the character embedding size and added four treebanks with very accurate taggers.

Predicted UPOS tags almost never beat a parser that uses no tags. Gold tags, by contrast, improve results by more than the trend across tagger accuracies would predict. The authors conclude that the hardest tags, the final few percentage points towards perfect accuracy, are the ones that matter most for parsing. In practice, predicted tags are only worthwhile with extremely accurate tagging. Practitioners should therefore test their usefulness for each language and system rather than including them by default.

Key points

  • Data and models: The study drew on nine UD v2.4 treebanks, covering Wolof (WTB), Uyghur (UDT), Tamil (TTB), Russian (GSD), Hebrew (HTB), Finnish (TDT), English (EWT), Chinese (GSD) and Ancient Greek (Perseus). Word embeddings came from fastText for most languages, Ginter et al. for Ancient Greek, and BPEmb for Wolof.
  • Parser sensitivity: UUParser draws more on POS features than Biaffine. With gold tags it gains roughly twice as much over the no-tag baseline.
  • Main exception: Tamil TTB with UUParser benefited from both gold and predicted tags. Tamil is the smallest treebank in the set and is agglutinative.
  • Character embeddings: Sizes of 32, 100, 180, 325 and 500 were tested. At every size, predicted tags still underperformed the no-tag baseline, so larger character embeddings did not change the overall finding.
  • Error analysis: ADV→ADJ and PROPN→ADJ were the only tagging errors significantly linked to parsing performance across all systems. CCONJ emerged as globally important. The authors suggest that a reduced subset of tags might work better than the full tag set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Dependency parsing · Universal Dependencies · Treebanks · Part-of-speech tagging

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.conll-1.6 · arXiv 2010.01830

@inproceedings{anderson2020frailty,
  title = {{On the Frailty of Universal POS Tags for Neural UD Parsers}},
  author = {Anderson, Mark and Gómez-Rodríguez, Carlos},
  booktitle = {Proceedings of the 24th Conference on Computational Natural Language Learning},
  year = {2020},
  doi = {10.18653/v1/2020.conll-1.6},
  eprint = {2010.01830},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.conll-1.6}
}

References

14 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramadan Alfared and Denis Béchet. 2012. POS taggers and dependency parsing. International Journal of Computational Linguistics and Applications, 3(2):107–122. find
  2. Yu Zhang, Zhenghua Li, Houquan Zhou, and Min Zhang. 2020. Is POS tagging necessary or even helpful for neural dependency parsing? arXiv preprint arXiv:2003.03204. — [arXiv:2003.03204](https://arxiv.org/abs/2003.03204) find
  3. Zuchao Li, Shexia He, Zhuosheng Zhang, and Hai Zhao. 2018. Joint learning of POS and dependencies for multilingual universal dependency parsing. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 65–73. find
  4. Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. arXiv preprint arXiv:1603.06598. find
  5. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2):159–191. find
  6. Dat Quoc Nguyen and Karin Verspoor. 2018. An improved neural network model for joint POS tagging and dependency parsing. find
  7. Joakim Nivre, Mitchell Abrams, Željko Agić, et al. 2019. Universal Dependencies 2.4. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  8. Vikas Raunak. 2017. Simple and effective dimensionality reduction for word embeddings. Proceedings of NIPS LLD Workshop. find
  9. Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82 treebanks, 34 models: Universal dependency parsing with multi-treebank models. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 113–123. find
  10. Aaron Smith, Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2018. An investigation of the interactions between pre-trained word embeddings, character models and POS tags in dependency parsing. In EMNLP 2018: 2018 Conference on Empirical Methods in Natural Language Processing, pages 2711– 2720. find
  11. Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019. Viable dependency parsing as sequence labeling. In NAACL-HLT 2019: Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 717– 723. find
  12. Atro Voutilainen. 1998. Does tagging help parsing? A case study on finite state parsing. In Proceedings of the International Workshop on Finite State Methods in Natural Language Processing, pages 25–36. Association for Computational Linguistics. find
  13. Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find
  14. Liner Yang, Meishan Zhang, Yang Liu, Maosong Sun, Nan Yu, and Guohong Fu. 2017. Joint POS tagging and dependence parsing with transition-based neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(8):1352– 1358. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/anderson-2020-frailty-universal-pos-tags-neural/