Elamur Research⌕
☰
home / papers / 2021 / A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing

A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing

Summary

The authors examine whether automatically predicted UPOS tags should be given to a neural dependency parser as features when annotated data are scarce. Previous studies have shown that on large treebanks, tags help only when the tagger is highly accurate. This paper investigates how that relationship changes with the amount of data. Three series of experiments are carried out with a BiLSTM-based biaffine parser. The first uses genuinely small UD v2.6 treebanks; there, the authors also compare against multi-task learning, in which tagging serves as an auxiliary task. The second uses artificially downsized samples from larger corpora, in which tagger accuracy was deliberately controlled. The third uses the smallest treebanks, augmented with synthetic trees produced by subtree swapping. Even moderately accurate tags turn out to raise LAS slightly. The effect is most noticeable when there are few full trees but considerably more POS-tagged sentences. As the amount of data grows, this advantage quickly disappears. For low-resource projects, this offers a practical guideline: part-of-speech annotation is cheaper, but new syntactic trees yield a larger gain.

Key points

  • Data: 12 UD v2.6 treebanks, each with fewer than 750 sentences. Very small, under 50 sentences: Buryat BDT, Kazakh KTB, Kurmanji MG, Livvi KKPP, Upper Sorbian UFAL. Small: Belarusian HSE, Galician TreeGal, Lithuanian HSE, Marathi UFAL, Old Russian RNC, Tamil TTB, Welsh CCG.
  • Small treebanks: average tagger accuracy is about 85.9%. Average LAS is 56.52 with predicted tags, 55.24 without tags, 64.13 with gold tags, and 55.10 in the multi-task setting. On the Lithuanian and Tamil treebanks, predicted tags worsen the result.
  • Very small treebanks: LAS is generally low, about 16. Gold tags nearly double it (30.39), while predicted tags add only about 0.6 points.
  • Artificial samples: Indonesian GSD, Irish IDT, Japanese GSD and Wolof WTB; sizes of 100, 232 and 541 sentences; tagger accuracy from 60 to 89%. Tags help most for a parser trained on 100 trees. However, increasing the number of trees from 100 to 232 yields more than 10 LAS points, whereas a tagger trained on 541 sentences yields only about 5.
  • Subtree-swapping augmentation: Kazakh, Kurmanji and Upper Sorbian; 10, 25 or 50 generated trees. It raises LAS by 3–6 points, but 25 and 50 trees give nearly identical results. More than 10 synthetic trees already harm the taggers. Pretrained language models were not considered in the paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Universal Dependencies · Treebanks · Part-of-speech tagging

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.iwpt-1.8 · arXiv 2106.04222

@inproceedings{anderson2021falta,
  title = {{A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing}},
  author = {Anderson, Mark and Dehouck, Mathieu and Gómez-Rodríguez, Carlos},
  booktitle = {Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021)},
  year = {2021},
  doi = {10.18653/v1/2021.iwpt-1.8},
  eprint = {2106.04222},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.iwpt-1.8}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić, Anders Johannsen, Barbara Plank, Héctor Martı́nez Alonso, Natalie Schluter, and Anders Søgaard. 2016. Multilingual projection for parsing truly low-resource languages. Transactions of the Association for Computational Linguistics, 4:301–312. find
  2. Dan Garrette, Jason Mielens, and Jason Baldridge. 2013. Real-world semi-supervised learning of pos-taggers for low-resource languages. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 583–592. find
  3. Mark Anderson and Carlos Gómez-Rodrı́guez. 2020. On the frailty of universal POS tags for neural UD parsers. In Proceeding of the 24th SIGNLL Conference on Computational Natural Language Learning, Online. find our card
  4. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(3):311–325. find
  5. Anders Johannsen, Željko Agić, and Anders Søgaard. 2016. Joint part-of-speech and dependency projection from multiple sources. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 561–566. find
  6. Jason Baldridge and Alexis Palmer. 2009. How well does active learning actually work? time-based evaluation of cost-reduction strategies for language documentation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 296–305. find
  7. Katharina Kann, Ophélie Lacroix, and Anders Søgaard. 2020. Weakly supervised POS taggers perform poorly on truly low-resource languages. Proceedings of the AAAI Conference on Artificial Intelligence, 34(5):8066–8073. find
  8. Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. arXiv preprint arXiv:1508.00657. find
  9. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 600–609. find
  10. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 62–72. find our card
  11. Aaron Smith, Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2018. An investigation of the interactions between pre-trained word embeddings, character models and POS tags in dependency parsing. arXiv preprint arXiv:1808.09060. find
  12. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan Mc-Donald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. Transactions of the Association for Computational Linguistics, 1:1–12. find
  13. Jörg Tiedemann. 2015. Cross-lingual dependency parsing with universal dependencies and predicted pos labels. Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 340–349. find our card
  14. Clara Vania, Yova Kementchedjhieva, Anders Søgaard, and Adam Lopez. 2019. A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Jo… find
  15. Daniel Zeman, Joakim Nivre, et al. 2020. Universal Dependencies 2.6. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  16. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages, pages 35– 42. find
  17. Yu Zhang, Zhenghua Li, Houquan Zhou, and Min Zhang. 2020. Is POS tagging necessary or even helpful for neural dependency parsing? arXiv preprint arXiv:2003.03204. — [arXiv:2003.03204](https://arxiv.org/abs/2003.03204) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anderson-2021-falta-de-pan-buenas-son/