Elamur Research⌕
☰
home / papers / 2021 / On the Relation between Syntactic Divergence and Zero-Shot Performance

On the Relation between Syntactic Divergence and Zero-Shot Performance

Summary

The authors test whether the quality of zero-shot syntactic parsing depends on how well syntactic relations are preserved in translation. Previous work examined this at the level of whole languages, whereas here the analysis is carried out for each individual edge of the dependency tree. A UD parser was trained on English only and applied to six languages: French, Russian, Japanese, Chinese, Korean and Arabic. For this purpose, they used a manually aligned portion of the PUD parallel corpus, about a thousand sentences. For Arabic, the corpus was extended separately. Each edge in a translation was assigned to one of several stability categories, depending on whether it has a counterpart in the English original. The authors then proposed three linguistically motivated transformations of UD that make the annotation more robust across languages. These were evaluated on a downstream task: zero-shot relation extraction on a portion of TACRED translated into Russian and Korean. The parser analyzes stable edges considerably more accurately, and the modified versions of UD mostly improve relation extraction. Hence, even modern multilingual models rely on the structural similarity of languages, and the annotation scheme can be made better suited for cross-lingual transfer.

Key points

  • Edges are divided into the categories Fully Aligned, Partially Aligned, Unaligned, Misaligned, Flipped and Function Word. By UAS, they almost always follow the order from Fully Aligned to Flipped. The most stable categories have roughly twice the LAS of the least stable ones. For example, for Russian this is LAS 83 versus 29, and for Chinese 48 versus 13. Results are averaged over 10 runs, and the standard deviation is usually below 2.
  • Partially Aligned edges have high UAS but low LAS. The parser tends to assign the label from the English sentence. This accounts for 46% of errors in the En-Ru pair and 42% in the En-Fr pair.
  • The three scheme transformations are called NOMINAL, PREDICATE and OBLIQUE. In the ensemble setting, NOMINAL raised F1 for Korean from 33 to 41.8. In the parallel ensemble setting, PREDICATE raised F1 for Russian from 38.1 to 46.1. In the standard setting for Russian, the gain is minimal and in some cases even negative.
  • Resources: 500 sentences from TACRED were translated into Korean and Russian, with relations transferred from the English original. A manually annotated and aligned extension of PUD for the En-Ar pair was also created. The materials are available on GitHub in the repositories OfirArviv/translated_tacred and OfirArviv/improving-ud.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.394 · arXiv 2110.04644

@inproceedings{arviv2021relation,
  title = {{On the Relation between Syntactic Divergence and Zero-Shot Performance}},
  author = {Arviv, Ofir and Nikolaev, Dmitry and Karidi, Taelin and Abend, Omri},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.394},
  eprint = {2110.04644},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.394}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018. Multitask parsing across semantic representations. In Proc. of ACL, pages 373–385. find
  2. Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia, Bulgaria. Association for Computational Linguistics. find our card
  3. Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, E. Hovy, Kai-Wei Chang, and Nanyun Peng. 2019. On difficulties of cross-lingual transfer with order differences: A case study on dependency parsing. In NAACL-HLT. find
  4. Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431–444. find our card
  5. Emanuele Bugliarello and Naoaki Okazaki. 2020. Enhancing machine translation with dependency-aware self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1618–1627, Online. Association for Computational Linguistics. find
  6. Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1936–1945, Vancouver, Canada. Assoc… find our card
  7. Kehai Chen, Rui Wang, M. Utiyama, E. Sumita, and T. Zhao. 2018. Syntax-directed attention for neural machine translation. In AAAI. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  9. Lu Liu, Yi Zhou, Jianhan Xu, Xiaoqing Zheng, Kai-Wei Chang, and Xuanjing Huang. 2020a. Cross-lingual dependency parsing by POS-guided word reordering. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2938– 2948, Online. Association for Computational Linguistics. find
  10. Nelson F. Liu, Daniel Hershcovich, Michael Kranzlein, and Nathan Schneider. 2020b. Lexical semantic recognition. ArXiv, abs/2004.15008. find
  11. François Maniez. 2012. A corpus-based study of adjectival vs nominal modification in medical english. In Alex Boulton, Shirley Carter-Thomas, and Elizabeth Rowley-Jolivet, editors, Corpus-Informed Research and Learning in ESP: Issues and Applications, pages 83–102. John Benjamins. find
  12. Marie-Catherine Marneffe, Bill MacCartney, and Christopher Manning. 2006. Generating typed dependency parses from phrase structure parses. volume 6. find our card
  13. Tao Meng, Nanyun Peng, and Kai-Wei Chang. 2019. Target language-aware constrained inference for cross-lingual dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP… find
  14. Amir Pouran Ben Veyseh Minh Van Nguyen, Viet Lai and Thien Huu Nguyen. 2021. Trankit: A lightweight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstra… find
  15. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019. Polyglot contextual representations improve crosslingual transfer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers… find
  16. Siva Reddy, Oscar Täckström, Slav Petrov, Mark Steedman, and Mirella Lapata. 2017. Universal semantic parsing. find
  17. Dmitry Nikolaev, Ofir Arviv, Taelin Karidi, Neta Kenneth, Veronika Mitnik, Lilja Maria Saeboe, and Omri Abend. 2020. Fine-grained analysis of cross-linguistic syntactic divergences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1159–1176, Online. A… find
  18. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  19. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth I… find our card
  20. Joakim Nivre, Paola Marongiu, Filip Ginter, Jenna Kanerva, Simonetta Montemagni, Sebastian Schuster, and Maria Simi. 2018. Enhancing Universal Dependency treebanks: A case study. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 102–107, Brussels, Belgium. Association… find our card
  21. Youngbin Noh, Jiyoon Han, Tae Hwan Oh, and Hansaem Kim. 2018. Enhancing universal dependencies for korean. In Proceedings of the second Workshop on Universal Dependencies (UDW 2018), pages 108–116. find
  22. Edoardo Maria Ponti, Roi Reichart, Anna Korhonen, and Ivan Vulić. 2018. Isomorphic transfer of syntactic structures in cross-lingual NLP. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1531–1542, Melbourne, Australia. Associ… find
  23. Ofek Rafaeli, Omri Abend, Leshem Choshen, and Dmitry Nikolaev. 2021. Part of speech and universal dependency effects on english arabic machine translation. find
  24. Mohammad Sadegh Rasooli and Michael Collins. 2019. Low-resource syntactic transfer with unsupervised source reordering. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),… find
  25. Sebastian Schuster and Christopher D. Manning. 2016. Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In LREC. find our card
  26. Gabriel Stanovsky, Jessica Ficler, Ido Dagan, and Yoav Goldberg. 2014. Intermediary semantic representation through proposition structures. In Proceedings of the ACL 2014 Workshop on Semantic Parsing, pages 66–70, Baltimore, MD. Association for Computational Linguistics. find
  27. Aryeh Tiktinsky, Yoav Goldberg, and Reut Tsarfaty. 2020. pyBART: Evidence-based syntactic transformations for IE. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 47–55, Online. Association for Computational Linguistics. find our card
  28. Dingquan Wang and Jason Eisner. 2018a. Surface statistics of an unknown language indicate how to parse it. Transactions of the Association for Computational Linguistics, 6:667–685. find
  29. Dingquan Wang and Jason Eisner. 2018b. Synthetic data made to order: The case of parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1325–1337, Brussels, Belgium. Association for Computational Linguistics. find
  30. Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  32. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics. find our card
  33. System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
  34. Haoran Xu and Philipp Koehn. 2021. Zero-shot cross-lingual dependency parsing through contextual embedding transformation. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 204–213, Kyiv, Ukraine. Association for Computational Linguistics. find
  35. Dian Yu, Heng Ji, Sujian Li, and Chin-Yew Lin. 2015. Why read if you can scan? trigger scoping strategy for biographical fact extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1203– 120… find
  36. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 35–45. find
  37. Daniel Zeman, Martin Popel, Milan Straka, Jan Hajič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdeňka Urešová… find
  38. CoNLL 2017 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics. find our card
  39. In this section, we give additional details on the results reported in the Edge Stability and ZS Parsability section (§2). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/arviv-2021-relation-between-syntactic-divergence-zero/