Elamur Research⌕
☰
home / papers / 2015 / Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

Summary

Dependency parsers that give each word type its own lookup embedding cannot share information across related surface forms. They also handle unseen words poorly, which is a serious weakness for morphologically rich languages. Ballesteros, Dyer and Smith extend the stack-LSTM transition-based parser of Dyer et al. (2015). Instead of word lookup vectors, they build each token's representation by reading its characters with a bidirectional LSTM in both directions and then combining the two final states with a POS tag embedding. The model receives no explicit morphological annotation. They also add the SWAP transition to the arc-standard system so that the parser can output nonprojective trees. Experiments on the SPMRL datasets, plus English and Chinese, show clear gains for agglutinative languages and those with rich case systems. Results remain competitive where morphology is sparse. In some languages, the character model even makes POS tags largely unnecessary. This suggests that treebank annotation budgets may be better spent on dependencies than on morphological features.

Key points

  • The base system is the stack-LSTM parser of Dyer et al. (2015). The buffer, stack and action history each have their own stack LSTM, and the parser greedily chooses actions using a log-linear model.
  • Character-based embeddings follow Ling et al. (2015). Forward and backward character LSTMs are concatenated with a tag vector and passed through a linear map with ReLU. The same mechanism covers both in-vocabulary and OOV words, so no "UNK" token is needed.
  • The SWAP operation (Nivre, 2009) enables nonprojective parsing. Because of it, the buffer may now hold partial tree fragments.
  • The largest improvements appear for Basque, Hungarian, Korean and Turkish. The authors report gains without pretrained embeddings, unlabeled data, explicit morphological features or parser ensembles.
  • Code for replicating the experiments is released at github.com/clab/lstm-parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dependency parsing · Part-of-speech tagging

Cited by

116 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 103 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1041 · arXiv 1508.00657

@inproceedings{ballesteros2015improved,
  title = {{Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs}},
  author = {Ballesteros, Miguel and Dyer, Chris and Smith, Noah A.},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1041},
  eprint = {1508.00657},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1041}
}

References

55 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne Abeillé, Lionel Clément, and François Toussenel. 2003. Building a treebank for French. In Treebanks. Springer. find
  2. Itziar Aduriz, Marı́a Jesús Aranzabe, Jose Mari Arriola, Aitziber Atutxa, Arantza Dı́az de Ilarraza, Aitzpea Garmendia, and Maite Oronoz. 2003. Construction of a Basque dependency treebank. In Proc of TLT. find
  3. Miguel Ballesteros. 2013. Effective morphological feature selection with maltoptimizer at the SPMRL 2013 shared task. In Proc. of SPMRL-EMNLP. find
  4. Anders Björkelund, Ozlem Cetinoglu, Richárd Farkas, Thomas Mueller, and Wolfgang Seeker. 2013. (Re)ranking Meets Morphosyntax: State-of-the-art Results from the SPMRL 2013 Shared Task. In SPMRL-EMNLP. find
  5. Anders Björkelund, Özlem Çetinoğlu, Agnieszka Faleńska, Richárd Farkas, Thomas Mueller, Wolfgang Seeker, and Zsolt Szántó. 2014. Introducing the IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 Shared Task: Reranking and Morpho-syntax meet Unlabeled Data. In SPMRL-SANCL. find
  6. Bernd Bohnet, Joakim Nivre, Igor Boguslavsky, Richard Farkas, Filip Ginter, and Jan Hajič. 2013. Joint morphological and syntactic analysis for richly inflected languages. TACL, 1. find
  7. Jan A. Botha and Phil Blunsom. 2014. Compositional Morphology for Word Representations and Language Modelling. In ICML. find
  8. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X. In Proc of CoNLL. find
  9. Danqi Chen and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proc. EMNLP. find our card
  10. Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. In Proc. IJCAI. find
  11. Jinho D. Choi. 2013. Preparing Korean Data for the Shared Task on Parsing Morphologically Rich Languages. ArXiv e-prints, September. find
  12. Grzegorz Chrupała. 2014. Normalizing tweets with edit scripts and recurrent neural embeddings. In Proc of ACL. find
  13. Shay B. Cohen and Noah A. Smith. 2007. Joint morphological and syntactic disambiguation. In Proc. EMNLP-CoNLL. find
  14. Cicero Nogueira dos Santos and Victor Guimarães. 2015. Boosting named entity recognition with neural character embeddings. Arxiv. find
  15. Cicero dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proc of ICML-14. find
  16. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proc of ACL. find our card
  17. Felix A. Gers, Nicol N. Schraudolph, and Jürgen Schmidhuber. 2002. Learning precise timing with LSTM recurrent networks. JMLR. find
  18. Yoav Goldberg and Michael Elhadad. 2011. Joint Hebrew segmentation and parsing using a PCFG-LA lattice parser. In Proc of ACL. find
  19. Yoav Goldberg and Joakim Nivre. 2013. Training deterministic parsers with non-deterministic oracles. TACL. find
  20. Yoav Goldberg and Reut Tsarfaty. 2008. A single generative model for joint morphological segmentation and syntactic parsing. In Proc of ACL. find
  21. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6). find
  22. Alex Graves. 2013. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850. find
  23. Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R. Steunebrink, and Jürgen Schmidhuber. 2015. LSTM: A search space odyssey. CoRR, abs/1503.04069. find
  24. Karl Moritz Hermann and Phil Blunsom. 2013. The role of syntax in vector space models of compositional semantics. In Proc. ACL. find
  25. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  26. Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, and David Sontag. 2010. Dual decomposition for parsing with non-projective head automata. In Proc of EMNLP. find
  27. Wang Ling, Tiago Luı́s, Luı́s Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proc. EMNLP. find our card
  28. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In NEMLAR Conference on Arabic Language Resources and Tools. find
  29. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19(2):313–330. find our card
  30. Marie-Catherine De Marneffe, Bill Maccartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proc of LREC. find our card
  31. Thomas Müller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morphological tagging. In Proc of EMNLP. find
  32. Joakim Nivre, Johan Hall, Jens Nilsson, Gülsen Eryiğit, and Svetoslav Marinov. 2006a. Labeled pseudo-projective dependency parsing with support vector machines. In Proc of CoNLL. find our card
  33. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006b. Talbanken05: A Swedish treebank with phrase structure and dependency annotation. In Proc of LREC, Genoa, Italy. find
  34. Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In Proc of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together. find our card
  35. Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In Proc of ACL. find our card
  36. Kemal Oflazer, Bilge Say, Dilek Zeynep Hakkani-Tür, and Gökhan Tür. 2003. Building a Turkish treebank. In Treebanks, pages 261–277. Springer. find
  37. Ferdinand Saussure. 1916. Nature of the linguistic sign. In Course in General Linguistics. find
  38. Djamé Seddah and Reut Tsarfaty. 2014. Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages. find
  39. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Gal-letebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yanni… find
  40. Wolfgang Seeker and Jonas Kuhn. 2012. Making Ellipses Explicit in Dependency Conversion for a German Treebank. In Proc of LREC. find
  41. Khalil Sima’an, Alon Itai, Yoad Winter, Alon Altman, and Noa Nativ. 2001. Building a Tree-Bank for Modern Hebrew Text. In Traitement Automatique des Langues. find
  42. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proc of NIPS. find
  43. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2013a. Grounded compositional semantics for finding and describing images with sentences. TACL. find our card
  44. Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc of EMNLP. find
  45. Pontus Stenetorp. 2013. Transition-based dependency parsing using recursive neural networks. In Proc of NIPS Deep Learning Workshop. find
  46. Marek Świdziński and Marcin Woliński. 2010. Towards a bank of constituent parse trees for Polish. In Proc of TSD. find
  47. Ivan. Titov and James. Henderson. 2007. A latent variable model for generative dependency parsing. In Proc of IWPT. find
  48. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proc of NAACL. find our card
  49. Reut Tsarfaty. 2006. Integrated morphological and syntactic disambiguation for Modern Hebrew. In Proc of ACL Student Research Workshop. find our card
  50. Veronika Vincze, Dóra Szauter, Attila Almási, György Móra, Zoltán Alexin, and János Csirik. 2010. Hungarian dependency treebank. In Proc of LREC. find
  51. David Weiss, Christopher Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In Proc of ACL. find
  52. Yue Zhang and Stephen Clark. 2008a. Joint word segmentation and POS tagging using a single perceptron. In Proc of ACL. find
  53. Yue Zhang and Stephen Clark. 2008b. A tale of two parsers: Investigating and combining graph-based and transition-based dependency parsing. In Proc of EMNLP. find our card
  54. Meishan Zhang, Yue Zhang, Wanxiang Che, and Ting Liu. 2013. Chinese parsing exploiting characters. In Proc of ACL. find
  55. Hao Zhou, Yue Zhang, Shujian Huang, and Jiajun Chen. 2015. A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing. In Proc of ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ballesteros-2015-improved-transition-based-parsing-modeling/