Elamur Research⌕
☰
home / papers / 2016 / Stack-propagation: Improved Representation Learning for Syntax

Stack-propagation: Improved Representation Learning for Syntax

Paper · 2016 · Yuan Zhang and David Weiss

Summary

Neural dependency parsers usually take POS tags from a separately trained tagger. Tagging mistakes then carry over into the parse, and the tagger cannot use syntactic context. Zhang and Weiss propose "stack-propagation", which treats POS supervision as a regularizer rather than an input feature.

A window-based feed-forward tagger and a greedy transition-based feed-forward parser, both in the style of Chen and Manning (2014) and Weiss et al. (2015), are linked directly. The parser's feature templates pull the tagger's hidden-layer activations for tokens on the stack and buffer, instead of discrete tags. Training alternates stochastic updates between the tagging and parsing objectives, and parser gradients flow back into the tagger. At test time, the parser needs no predicted tags.

On 19 Universal Dependencies languages, the model improves labeled accuracy by 1.3% absolute over a strong graph-based parser and by 2.7% over a character-LSTM greedy parser. It is also smaller and faster. The work shows that upstream annotations can shape learned representations instead of acting as fragile pipeline inputs.

Key points

  • Evaluation on 19 UD treebanks: +1.3% absolute over the graph-based parser of Lei et al. (2014) and +2.7% over the greedy LSTM parser of Ballesteros et al. (2015).
  • Cascaded errors drop sharply. On tokens the pipeline tagger mislabels, LAS rises from 34.1% to 45.0% (+10.9%), compared with a +1.8% gain on all other tokens.
  • Because the parser and tagger share parameters, the combined model has roughly half the parameters of the pipeline baseline and runs about twice as fast.
  • On the WSJ corpus, stackprop performs about as well with the coarse universal tagset as with fine tags, while the pipeline baseline degrades strongly. It beats earlier greedy parsers such as Dyer et al. (2015).
  • Stackprop outperforms joint POS+parsing modeling with the transition system of Bohnet and Nivre (2012). Combining stackprop with the joint system improves tagging but slightly lowers parsing accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Syntax & grammar formalisms · 214 works

Cite

Cite as Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved Representation Learning for Syntax. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1147 · arXiv 1603.06598

@inproceedings{zhang2016stack,
  title = {{Stack-propagation: Improved Representation Learning for Syntax}},
  author = {Yuan Zhang and David Weiss},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1147},
  eprint = {1603.06598},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1147}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010, pages 177–186. Springer. find
  2. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 1, pages 740–750.
  3. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. JMLR.
  4. Marie-Catherine De Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of Fifth International Conference on Language Resources and Evaluation, pages 449– 454.
  5. Greg Durrett and Dan Klein. 2015. Neural crf parsing. In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics. find
  6. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. ACL.
  7. Nicholas FitzGerald, Oscar Tckstrm, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP ’15). find
  8. Jun Hatori, Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2011. Incremental joint pos tagging and dependency parsing in chinese. In IJC-NLP, pages 1216–1224. Citeseer. find
  9. Geoffrey E Hinton. 2012. A practical guide to training restricted boltzmann machines. In Neural Networks: Tricks of the Trade, pages 599–619. Springer. find
  10. Chris Alberti, David Weiss, Greg Coppola, and Slav Petrov. 2015. Improved transition-based parsing and tagging with neural networks. In Proceedings of EMNLP 2015. find
  11. Terry Koo, Xavier Carreras Pérez, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In 46th Annual Meeting of the Association for Computational Linguistics, pages 595–603.
  12. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. Proceddings of EMNLP.
  13. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1381–1391. find
  14. Bernd Bohnet and Joakim Nivre. 2012. A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1455–1465.…
  15. Zhenghua Li, Min Zhang, Wanxiang Che, Ting Liu, Wenliang Chen, and Haizhou Li. 2011. Joint models for chinese pos tagging and dependency parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1180–1191. Association for Computational Linguistics. find
  16. Zhenghua Li, Min Zhang, Wanxiang Che, Ting Liu, and Wenliang Chen. 2014. Joint optimization for chinese POS tagging and dependency parsing. IEEE/ACM Transactions on Audio, Speech & Language Processing, pages 274–286. find
  17. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330.
  18. Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together, IncrementParsing ’04, pages 50–57, Stroudsburg, PA, USA. Association for Computational Linguistics.
  19. Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 351–359.
  20. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086.
  21. Xian Qian and Yang Liu. 2012. Joint chinese word segmentation, pos tagging and parsing. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 501– 511. Association for Computational Linguistics. find
  22. Zhiguo Wang and Nianwen Xue. 2014. Joint pos tagging and transition-based constituent parsing in chinese with non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 733–742. find
  23. David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. In Proceedings of ACL 2015, pages 323–333. find
  24. David H Wolpert. 1992. Stacked generalization. Neural networks. find
  25. Yuan Zhang, Chengtao Li, Regina Barzilay, and Kareem Darwish. 2015. Randomized greedy inference for joint segmentation, POS tagging and dependency parsing. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics, pages 42–52. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2016-stack-propagation-improved-representation-learning/