Elamur Research⌕
☰
home / papers / 2016 / What Do Recurrent Neural Network Grammars Learn About Syntax?

What Do Recurrent Neural Network Grammars Learn About Syntax?

Paper · 2016 · Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong et al.

Summary

Neural parsers perform well, but it is hard to tell what syntactic knowledge they actually acquire. This paper treats recurrent neural network grammars (RNNGs) as objects of linguistic inquiry. RNNGs are generative models that build phrase-structure trees top-down using a stack, a buffer and an action history. On the English PTB (§02–21 train, §24 dev, §23 test), the authors train variants with components removed. They evaluate these on constituency parsing, on dependency parsing after conversion to Stanford dependencies, and on language modeling. They then add a gated attention mechanism to the composition function (GA-RNNG) to reveal which children a phrase vector depends on. A label-free version (U-GA-RNNG) tests how much nonterminal categories matter. Three findings emerge:

  • Explicit composition on the stack is the decisive ingredient.
  • Attention largely recovers lexical heads, mostly agreeing with hand-written head rules but sometimes spreading weight over several children, e.g. in conjunctions.
  • Phrase vectors learned without labels still cluster into traditional categories.

These results support the endocentricity hypothesis and suggest that bracketing carries most of the syntactic information.

Key points

  • Removing the stack hurts most: PTB §23 F1 falls from 93.3 (baseline RNNG) to 92.5. Dropping the buffer or the action history barely matters.
  • A stack-only RNNG without ensembling reached 93.6 F1 on PTB. After conversion, it achieved 95.8 UAS and 94.6 LAS on PTB SD, the best supervised results reported at the time. GA-RNNG scored 93.5 F1.
  • RNNGs beat linearized-tree models: 91.2 vs. 88.3 F1 against Vinyals et al. (2015) in the discriminative setting, and 93.3 vs. 92.6 against Choe and Charniak (2016) in the generative setting.
  • In U-GA-RNNG, PP and SBAR phrases opening with preposition-like words cluster together. Phrases built on of and because form their own groups.
  • Code and pretrained models are released at github.com/clab/rnng (interpreting-rnng).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓University of WashingtonSeattle · United States402 works · 212,763 citations🏢IBMArmonk · United States154 works · 72,606 citations🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States55 works · 49,277 citations🎓University of PittsburghPittsburgh · United States65 works · 23,401 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksNeuroscience · 186 worksCognitive Neuroscience · 185 works

🧭 Topics 1Research subjects and languages the work is about.Syntax & grammar formalisms · 221 works

Cite

Cite as Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig and Noah A. Smith. 2016. What Do Recurrent Neural Network Grammars Learn About Syntax?. arXiv:1611.05774.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.05774 · arXiv 1611.05774

@misc{kuncoro2016what,
  title = {{What Do Recurrent Neural Network Grammars Learn About Syntax?}},
  author = {Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig and Noah A. Smith},
  year = {2016},
  doi = {10.48550/arxiv.1611.05774},
  eprint = {1611.05774},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.05774}
}

References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In Proc. of ACL.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
  3. David Chiang and Daniel M. Bikel. 2002. Recovering latent information in treebanks. In Proc. of COL-ING. find
  4. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proc. of EMNLP. find
  5. Noam Chomsky. 1970. Remarks on nominalization. In Readings in English Transformational Grammar. find
  6. Noam Chomsky. 1993. A Minimalist Program for Linguistic Theory. find
  7. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proc. of EACL.
  8. Marie-Catherine De Marneffe and Christopher D. Manning. 2008. Stanford typed dependencies manual. find
  9. Marie-Catherine De Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proc. of LREC.
  10. Timothy Dozat and Christopher D. Manning. 2016. Deep biaffine attention for neural dependency parsing. arXiv:1611.01734.
  11. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proc. of NAACL. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. find
  13. Ray Jackendoff. 1977. X’ Syntax. find
  14. T. Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology, 61(1). find
  15. Janne Bondi Johannessen. 1998. Coordination. find
  16. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In Proc. of NODALIDA.
  17. Mark Johnson. 1998. PCFG models of linguistic tree representations. Computational Linguistics.
  18. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and understanding recurrent networks. arXiv:1506.02078.
  19. Edward L. Keenan. 1987. Multiply-headed noun phrases. Linguistic Inquiry, 18(3):481–490. find
  20. Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. TACL.
  21. Dan Klein and Christopher D. Manning. 2002. A generative constituent-context model for improved grammar induction. In Proc. of ACL.
  22. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proc. of ACL.
  23. Lingpeng Kong and Noah A. Smith. 2014. An empirical comparison of parsing methods for stanford dependencies. arXiv:1404.4314.
  24. Jiwei Li, Dan Jurafsky, and Eduard Hovy. 2015. When are tree structures necessary for deep learning of representations? In Proc. of EMNLP.
  25. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proc. of NAACL.
  26. Tom M. Mitchell. 1980. The need for biases in learning generalizations. find
  27. Fernando Pereira and Yves Schabes. 1992. Inside-outside reestimation from partially bracketed corpora. In Proc. of ACL.
  28. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proc. of COLING-ACL.
  29. Federico Sangati and Willem Zuidema. 2009. Unsupervised methods for head assignments. In Proc. of EACL. find
  30. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proc. of EMNLP.
  31. Laurens van der Maaten and Geoffrey E. Hinton. 2008. Visualizing high-dimensional data using tsne. Journal of Machine Learning Research, 9. find
  32. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In NIPS. find
  33. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proc. of EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuncoro-2016-recurrent-neural-network-grammars-learn/