Elamur Research⌕
☰
home / papers / 2023 / Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency Parsing

Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency Parsing

Paper · 2023 · Rochelle Choenni, Dan Garrette and Ekaterina Shutova

Summary

Multilingual encoders like mBERT use one set of parameters for all languages. This enables transfer, but updates from dissimilar languages can conflict, and low-resource languages suffer most. The authors manage parameter sharing during fine-tuning with language-specific subnetworks. For each training language, they fine-tune the model and prune away the least useful attention heads, producing a binary head mask. Masks can overlap, so related languages share heads while distant ones stay separated. Besides fixed (static) masks, they introduce dynamic masks that change together with the model during training, and they integrate both variants into MAML meta-learning. The parser follows the UDify design (mBERT plus a deep biaffine decoder). It is trained on 8 high-resource UD treebanks and evaluated, after few-shot adaptation, on 82 UD datasets. Combining meta-learning with dynamic subnetworks gave the best cross-lingual transfer to unseen languages. The work shows that data-driven, structured sharing can limit negative interference without adding new parameters. Its analyses also suggest which method suits which kind of target language.

Key points

  • Evaluation covers 82 UD datasets in 70 languages from 43 families, and 58 of these languages are treated as truly low-resource.
  • The 8 training treebanks are PADT (ar), PDT (cs), EWT (en), HDTB (hi), ISDT (it), EDT (et), Norsk (no) and SynTag (ru). A second set of 7 languages (be, fi, ga, he, id, tr, zh) was used to check robustness.
  • Subnetworks are learned at the attention-head level. Static and dynamic variants are compared, with and without MAML, following the setup of Langedijk et al. (2022).
  • Analyses examine typological relatedness, scarcity of pretraining data, domain shift, and the prediction of rare or unseen dependency labels.
  • Limitations: because the method relies on mBERT, it cannot handle scripts unseen in pretraining. Meta-learning is also sensitive to hyperparameters, which were tuned on Bulgarian, Japanese, Telugu and Persian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksDependency parsing · 131 worksLow-resource languages · 267 works

Cite

Cite as Rochelle Choenni, Dan Garrette and Ekaterina Shutova. 2023. Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency Parsing. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00482

@article{choenni2023cross,
  title = {{Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency Parsing}},
  author = {Rochelle Choenni, Dan Garrette and Ekaterina Shutova},
  journal = {Computational Linguistics},
  year = {2023},
  doi = {10.1162/coli\_a\_00482},
  url = {https://doi.org/10.1162/coli_a_00482}
}

References

24 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ansell, Alan, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4762–4781. https://doi…
  2. Arivazhagan, Naveen, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. In Proceedings of the 2019 Conference of the North Am…
  3. Bartoldson, Brian, Ari Morcos, Adrian Barbu, and Gordon Erlebacher. 2020. The generalization-stability tradeoff in neural network pruning. In Advances in Neural Information Processing Systems, volume 33, pages 20852–20864. Bengio, Yoshua, Nicholas Léonard, and Aaron Courville. 2013. Estimating or p… find
  4. Chi, Ethan A., John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564–5577. https://doi.org/10.18653/v1/2020 .acl-main.493 Choenni, Rochelle an… ↗
  5. Choudhary, Chinmay. 2021. Improving the performance of UDify with linguistic typology knowledge. In Proceedings of the Third Workshop on Computational Typology and Multilingual NLP, pages 38–60. https://doi.org/10.18653/v1/2021 .sigtyp-1.5 Chu, Yoeng Jin. 1965. On the shortest arborescence of a dire… ↗
  6. Clark, Kevin, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT Look at? An analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286. https://doi.org/10 .18653/v1/W19-4828 Devlin… find
  7. Dozat, Timothy and Christopher D. Manning. 2016. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734. https://doi.org/10.48550/arXiv.1611.01734 Finn, Chelsea, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks… ↗
  8. Foroutan, Negar, Mohammadreza Banaei, Remi Lebret, Antoine Bosselut, and Karl Aberer. 2022. Discovering language-neutral sub-networks in multilingual language models. arXiv preprint arXiv:2205.12672. find
  9. Frankle, Jonathan and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. Glavaš, Goran and Ivan Vulić. 2021. Climbing the tower of treebanks: Improving low-resource dependency parsing via hierarch…
  10. Han, Song, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, volume 28, pages 1135–1143. find
  11. Held, William and Diyi Yang. 2022. Shapley head pruning: Identifying and removing interference in multilingual transformers. arXiv preprint arXiv:2210.05709. https://doi.org/10.48550/arXiv.2210.05709 Hendy, Amr, Mohamed Abdelghaffar, Mohamed Afify, and Ahmed Y. Tawfik. 2022. Domain specific sub-net… ↗
  12. Houlsby, Neil, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pages 2790–2799. find
  13. Htut, Phu Mon, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in BERT track syntactic dependencies? arXiv preprint arXiv:1911.12246. https://doi.org/10.48550/arXiv.1911.12246 Jiao, Xiaoqi, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.… ↗
  14. Kondratyuk, Dan and Milan Straka. 2019. 75 languages, 1 model: Parsing universal dependencies universally. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2779–27…
  15. Langedijk, Anna, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, and Ekaterina Shutova. 2022. Meta-learning for fast cross-lingual adaptation in dependency parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volu…
  16. Li, Hao, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convNets. arXiv preprint arXiv:1608.08710. — [arXiv:1608.08710](https://arxiv.org/abs/1608.08710) find
  17. Li, Yanyang, Fuli Luo, Runxin Xu, Songfang Huang, Fei Huang, and Liwei Wang. 2022. Probing structured pruning on multilingual pre-trained models: Settings, algorithms, and efficiency. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  18. Littell, Patrick, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and Lang2Vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational…
  19. Naseem, Tahira, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 629–637.
  20. Nivre, Joakim, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on…
  21. Nooralahzadeh, Farhad, Ioannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4547–4562. https://doi.org/10.18653/v1/2020.emnlp -main.368 Nooralahzadeh,… ↗
  22. Voita, Elena, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808. https://doi.org/10.18653/v1/P… ↗
  23. Zeman, Daniel, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. find
  24. CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual parsing from raw text to universal dependencies, pages 1–21. https://doi.org/10.18653/v1/K18-2001 Zhang, Zhengyan, Fanchao Qi, Zhiyuan Liu, Qun Liu, and M… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choenni-2023-cross-lingual-transfer-language-specific/