Elamur Research⌕
☰
home / papers / 2015 / Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing

Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing

Paper · 2015 · Yuan Zhang and Regina Barzilay

Summary

Transfer parsing builds a dependency parser for a target language that has little or no annotation by reusing treebanks from other languages. The strongest such systems rely on hand-built feature templates. These templates join standard arc features with typological properties, such as WALS word-order codes. Low-rank tensor models can learn feature conjunctions automatically, but they combine all features freely. That includes meaningless pairings, like two different head POS values together with a verb-subject ordering feature.

Zhang and Barzilay propose a hierarchical low-rank tensor to fix this. It composes atomic features step by step into intermediate embeddings:

  • head, modifier and direction/length first;
  • then the arc label.

Expert typological features are added at the level whose atomic features they describe. They are summed rather than multiplied, so invalid interactions are excluded by design. Algebraically, the model is a sum of ordinary tensors with shared components, so standard online training still works.

The model was tested on the universal dependency treebank for 10 languages, with labeled parsing included. It outperforms the selective-sharing parser of Täckström et al. (2013), direct transfer and a plain multiway tensor. This holds in both unsupervised and semi-supervised transfer.

Key points

  • With source languages only, the model beats the baselines on both UAS and LAS for 7 of 10 languages. Its average UAS gain is 1.1% over Täckström et al. (2013) and 4.8% over direct transfer (McDonald et al., 2011).
  • With 50 annotated target sentences added, the UAS gains rise to 1.7% over NT-Select and 4.5% over direct transfer. Average scores reach 77.9 UAS and 68.5 LAS.
  • Compared with a traditional multiway tensor, which gives non-zero weights even to invalid combinations, the hierarchical version improves unsupervised UAS by 0.5% and LAS by 1.3%.
  • Against RBGParser trained on full treebanks with partial lexicalization (82.9 UAS / 74.5 LAS), transfer closes about 30% of the gap left by training on 50 sentences alone.
  • Decoding runs at about the speed of the multiway tensor (58.6 vs. 61.2 sentences/sec on English). Code is available at github.com/yuanzh/TensorTransfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMathematics · 177 worksComputational Mathematics · 2 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksEfficient models & compression · 242 works

Cite

Cite as Yuan Zhang and Regina Barzilay. 2015. Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1213

@inproceedings{zhang2015hierarchical,
  title = {{Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing}},
  author = {Yuan Zhang and Regina Barzilay},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1213},
  url = {https://doi.org/10.18653/v1/d15-1213}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Taylor Berg-Kirkpatrick and Dan Klein. 2010. Phylogenetic grammar induction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1288–1297. Association for Computational Linguistics. find
  2. Shay B Cohen and Noah A Smith. 2009. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 74–82. Association for Computational Lin…
  3. Shay B Cohen, Dipanjan Das, and Noah A Smith. 2011. Unsupervised structure prediction with non-parallel multilingual guidance. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 50–61. Association for Computational Linguistics.
  4. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online passive-aggressive algorithms. The Journal of Machine Learning Research, 7:551–585. find
  5. Marie-Catherine De Marneffe and Christopher D Manning. 2008. The stanford typed dependencies representation. In Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, pages 1–8. Association for Computational Linguistics.
  6. Matthew S Dryer, David Gil, Bernard Comrie, Hagen Jung, Claudia Schmidt, et al. 2005. The world atlas of language structures. find
  7. Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Cross-lingual transfer for unsupervised dependency parsing without parallel data. Proceedings of the SIGNLL Conference on Computational Natural Language Learning, page 113. find
  8. Greg Durrett, Adam Pauls, and Dan Klein. 2012. Syntactic transfer using a bilingual lexicon. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1–11. Association for Computational Linguistics. find
  9. Jason M Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In Proceedings of the 16th conference on Computational linguistics-Volume 1, pages 340–345. Association for Computational Linguistics.
  10. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the Annual Conference of the European Chapter of the Association for Computational Linguistics., volume 2014.
  11. Daniel Fried, Tamara Polajnar, and Stephen Clark. 2015. Low-rank tensors for verbs in compositional distributional semantics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. find
  12. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural language engineering, 11(03):311–325. find
  13. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1381–1391. find
  14. Tao Lei, Yuan Zhang, Regina Barzilay, Lluı́s Màrquez, and Alessandro Moschitti. 2015. High-order low-rank tensors for semantic role labeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
  15. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 91–98. Association for Computational Linguistics.
  16. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 62–72. Association for Computational Linguistics.
  17. Ryan T McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith B Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Comp…
  18. Tahira Naseem, Harr Chen, Regina Barzilay, and Mark Johnson. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1234–1244. Association for Computational Linguistics.
  19. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 629–637. Association for Computational Linguistics.
  20. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086.
  21. Audi Primadhanty, Xavier Carreras, and Ariadna Quattoni. 2015. Low-rank regularization for sparse conjunctive feature spaces: An application to named entity classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. find
  22. Ariadna Quattoni, Borja Balle, Xavier Carreras, and Amir Globerson. 2014. Spectral regularization for max-margin sequence tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1710–1718. find
  23. Mohammad Sadegh Rasooli and Michael Collins. 2015. Density-driven cross-lingual transfer of dependency parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  24. Sameer Singh, Tim Rocktaschel, and Sebastian Riedel. 2015. Towards combined matrix and tensor factorization for universal schema relation extraction. In NAACL Workshop on Vector Space Modeling for NLP. find
  25. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of the Advances in Neural Information Processing Systems, pages 926–934.
  26. Vivek Srikumar and Christopher D Manning. 2014. Learning distributed representations for structured output prediction. In Proceedings of the Advances in Neural Information Processing Systems, pages 3266–3274. find
  27. Oscar Täckström, Ryan McDonald, and Joakim Nivre. 2013. Target language adaptation of discriminative transfer parsers. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics.
  28. Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. Transactions of the Association for Computational Linguistics, 3:227– 242. find
  29. Dong Yu, Li Deng, and Frank Seide. 2013a. The deep tensor neural network with applications to large vocabulary speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 21(2):388–396. find
  30. Mo Yu, Tiejun Zhao, Yalong Bai, Hao Tian, and Dianhai Yu. 2013b. Cross-lingual projections between languages from different families. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 312–317. find
  31. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In Proceedings of the International Joint Conference on Natural Language Processing, pages 35–42.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2015-hierarchical-low-rank-tensors-multilingual/