Elamur Research
☰
home / papers / 2016 / Many Languages, One Parser

Many Languages, One Parser

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q16-1031.pdf

Summary

Most multilingual NLP pipelines train and maintain a separate model for each language. This ignores structure that related languages share and makes it costly to deploy tools across many languages. Ammar et al. replace that collection with a single dependency parser, MALOPA, trained on the pooled treebanks of several languages. It builds on the stack LSTM (S-LSTM) transition parser of Dyer et al. (2015). Each token's input combines two kinds of signal:

  • Shared across languages: coarse universal POS tags, multilingual word embeddings and Brown-style word clusters, which let knowledge transfer between languages.
  • Specific to a language: a language embedding derived from typology and fine-grained POS tags, which let the parser adjust its behavior to the current language.

The authors tested seven European languages from UDT 2.0 and UD 1.2 in three settings: a large, a small, or no target treebank. With large treebanks, the single model matched or beat monolingual parsers on average. Its advantage was largest in the low-resource settings. The work shows that one shared model can serve many languages without losing accuracy on well-resourced ones.

Key points

  • The languages were German, English, Spanish, French, Italian, Portuguese and Swedish, drawn from Universal Dependency Treebanks v2.0 and UD v1.2. The code is released at github.com/clab/language-universal-parser.
  • When each target language had only about 3,000 annotated tokens, MALOPA beat a strong monolingual baseline by an average of 5.7 LAS points.
  • In zero-treebank transfer, the parser was trained on the other six UDT 2.0 languages using gold coarse POS tags. It averaged 70.5 LAS, compared with 69.3 for Guo et al. (2016) and 65.9 for Zhang and Barzilay (2015). It achieved the top score on four of the six target languages.
  • Language embeddings were left out of the zero-treebank setting because preliminary tests showed they hurt accuracy there.
  • The paper also studies how automatic language identification affects multilingual parsing quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Treebanks · Part-of-speech tagging

Cited by

150 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 139 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00109

@article{ammar2016many,
  title = {{Many Languages, One Parser}},
  author = {Ammar, Waleed and Mulcaire, George and Ballesteros, Miguel and Dyer, Chris and Smith, Noah A.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00109},
  url = {https://doi.org/10.1162/tacl_a_00109}
}

References

50 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić, Maria Jesus Aranzabe, Aitziber Atutxa, Cristina Bosco, Jinho Choi, Marie-Catherine de Marneffe, Timothy Dozat, Richárd Farkas, Jennifer Foster, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Yoav Goldberg, Jan Hajič, Anders Trærup Johannsen, Jenna Kanerva, Juha Kuokkala, Veronika Laippa… find
  2. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016. Massively multilingual word embeddings. arXiv:1602.01925v2. find our card
  3. Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. In EMNLP Workshop on Computational Approaches to Code Switching. find
  4. Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology, 6(3):1–26. find
  5. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proc. of CoNLL. find our card
  6. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proc. of EMNLP. find our card
  7. Shay B. Cohen, Dipanjan Das, and Noah A. Smith. 2011. Unsupervised structure prediction with non-parallel multilingual guidance. In Proc. of EMNLP. find
  8. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  9. Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015a. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In Proc. of ACL-IJCNLP. find
  10. Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015b. A neural network model for low-resource universal dependency parsing. In Proc. of EMNLP. find
  11. Greg Durrett, Adam Pauls, and Dan Klein. 2012. Syntactic transfer using a bilingual lexicon. In Proc. of EMNLP. find
  12. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proc. of ACL. find our card
  13. Matt Gardner, Kejun Huang, Evangelos Papalexakis, Xiao Fu, Partha Talukdar, Christos Faloutsos, Nicholas Sidiropoulos, and Tom Mitchell. 2015. Translation invariant word embeddings. In Proc. of EMNLP. find
  14. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. of AISTATS. find
  15. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proc. of ICASSP. find
  16. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proc. of ACL. find
  17. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proc. of AAAI. find
  18. Ulrich Heid and Sybille Raab. 1989. Collocations in multilingual generation. In Proc. of EACL. find
  19. James Henderson and Ivan Titov. 2010. Incremental sigmoid belief networks for grammar learning. Journal of Machine Learning Research, 11:3541–3570. find
  20. James Henderson. 2003. Inducing history representations for broad coverage statistical parsing. In Proc. of NAACL-HLT. find
  21. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv:1508.01991. find our card
  22. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(03):311–325. find
  23. Mohit Iyyer, Varun Manjunatha, Jordan L. Boyd-Graber, and Hal Daumé. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. of ACL. find
  24. Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proc. of ACL. find our card
  25. Xuezhe Ma and Fei Xia. 2014. Unsupervised dependency parsing with transferring distribution via parallel guidance and entropy regularization. In Proc. of ACL. find
  26. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proc. of EMNLP. find our card
  27. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proc. of ACL. find
  28. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proc. of ICLR. find
  29. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proc. of ACL. find
  30. Joakim Nivre and Jens Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. of ACL. find
  31. Joakim Nivre, Johan Hall, Sandra Kubler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
  32. The CoNLL 2007 shared task on dependency parsing. In Proc. of CoNLL. Joakim Nivre, Željko Agić, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Cristina Bosco, Sam Bowman, Giuseppe G. A. Celano, Miriam Connor, Marie-Catheri… find
  33. Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together. find our card
  34. Robert Östling. 2015. Word order typology through multilingual word alignment. In Proc. of ACL-IJCNLP. find
  35. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proc. of LREC. find
  36. Mohammad Sadegh Rasooli and Michael Collins. 2015. Density-driven cross-lingual transfer of dependency parsers. In Proc. of EMNLP. find
  37. Deitmar Rösner. 1988. The generation system of the SEMSYN project: Towards a task-independent generator for German. Advances in Natural Language Generation, 2. Anders Søgaard, Željko Agić, Héctor Martínez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. 2015. Inverted indexing for cross-l… find
  38. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  39. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  40. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proc. of NAACL-HLT. find our card
  41. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan Mc-Donald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. Transactions of the Association for Computational Linguistics, 1:1–12. find
  42. Jörg Tiedemann, Zeljko Agic, and Joakim Nivre. 2014. Treebank translation for cross-lingual parser induction. In Proc. of CoNLL. find
  43. Jörg Tiedemann. 2015. Cross-lingual dependency parsing with universal dependencies and predicted POS labels. In Proc. of International Conference on Dependency Linguistics (Depling). find
  44. Ivan Titov and James Henderson. 2007. Constituent parsing with incremental sigmoid belief networks. In Proc. of ACL. find
  45. David Vilares, Carlos Gómez-Rodríguez, and Miguel A. Alonso. 2016. One model, two languages: Training bilingual parsers with harmonized treebanks. arXiv:1507.08449v2. find
  46. Wikipedia. 2016. List of languages by number of native speakers. http://bit.ly/1LUP5kJ. Accessed: 2016-01-26. ↗
  47. Min Xiao and Yuhong Guo. 2014. Distributed word representation learning for cross-lingual dependency parsing. In Proc. of CoNLL. find
  48. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proc. of HLT. find
  49. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In Proc. of IJCNLP. find
  50. Yuan Zhang and Regina Barzilay. 2015. Hierarchical low-rank tensors for multilingual transfer parsing. In Proc. of EMNLP. find